7         Stage 2: Cleaning A Set of Data

Once you have a rogue mask that you are satisfied with, IRSCLEAN moves on to "Stage 2", (unless you have set the keyword /noStage2). Now you will be given the chance to apply the mask to other data files from the same IRS array. NOTE: Since rogue pixels can vary on timescales of hours, the effectiveness of a rogue mask for cleaning data is usually limited to data from the same AOR used to derive the mask.

 

NOTE: The pixel cleaning algorithm is found in the IDL function imbadfix.pro, which is part of the IRSCLEAN distribution. Pixel cleaning is accomplished by deriving profiles for the 3 rows above and below the bad pixel, predicting the profile for the bad pixel's row, scaling that profile by the good pixels in the bad pixel's row, and substituting  the value for the pixel in question from the predicted and scaled profile.

7.1       Picking Files to Clean (filesToClean)

NOTE: If you want to skip Stage 2 but forgot to set /noStage2, you can quit by choosing "Cancel" when asked to choose files to clean.

 

If you have not set the filesToClean keyword, IRSCLEAN opens a file picker dialog, which you can use to pick any (or all) of the following:

 

   1. One or more .fits files.

   2. One or more .txt (ASCII text) files that list a set of fits files, one per line.

 

An example .txt file is as follows:

 

#  an example .txt file

SPITZER_S3_13349376_0012_0000_8_bcd.fits

SPITZER_S3_13349376_0012_0001_8_bcd.fits

SPITZER_S3_13349376_0012_0002_8_bcd.fits

SPITZER_S3_13349376_0012_0003_8_bcd.fits

SPITZER_S3_13349376_0013_0000_8_bcd.fits

SPITZER_S3_13349376_0013_0001_8_bcd.fits

SPITZER_S3_13349376_0013_0002_8_bcd.fits

SPITZER_S3_13349376_0013_0003_8_bcd.fits

 

NOTE: Comments may be embedded in .txt files by putting the pound sign (#) at the beginning of a line.

 

Figure 7.1 Choosing files to clean (Stage 2).

 

The program is flexible enough to allow you to choose a combination of .fits and .txt files at the same time. The files will be parsed to create a master list of all .fits files to be processed. For Unix-type operating systems (including Mac OS X), multiple files can be selected by holding down the ‘Control’ key and clicking on the files. An example of a user choosing the sample files listed above is displayed in Figure 7.1.

 

The file selection process is subject to the following rules:

• All .fits and .txt files chosen via the dialogue must be in the same directory. However, .fits files listed inside a .txt file can come from anywhere, so long as the full path to the file is specified inside the .txt file.
• For .fits files listed inside a .txt file, the default path is the source directory of the .txt file. If you want to use a different directory, you should specify it explicitly in the entry.
• If you accidentally choose the same file (full path) more than once, it will only be processed once. On the other hand, files with the same base-names that reside in different directories are not considered duplicates and will be processed separately.

 

To skip interactive file-picking, set the filesToClean keyword at the command line. For example,

 

IDL> irsclean_mask, $

'sampledata/r13349376/ch3/bcd/SPITZER_S3_13349376_0012_0000_8_bcd.fits', $

inRmask_File='sampledata/r13349376/ch3/bcd/SPITZER_S3_13349376_0012_0000_8_rmask.fits',/noMaskEdit,/autoSave,$

filesToClean='sampledata/r13349376/ch3/bcd/SPITZER_S3_13349376_clean_batch_list.txt'

 

If the path is not given explicitly, files are assumed to be in the current working directory.

 

Figure 7.2 Editing the list of files to clean (Stage 2).

7.2       Reviewing Files to Clean

After all .fits file names have been extracted from the input .txt files and combined with any .fits files given explicitly, the parsed list will be saved to the file clean_batch_list.txt in the same directory as dataFile. Then IRSCLEAN will open a file-editing dialog, so you can review and edit (if necessary) the complete list. An example editing window is shown above (previous page).

 

For your convenience, the full path to each file will be listed, but you are allowed to add files to the list without giving the path, provided the file is in the current working directory.  (File names have been edited in the display above to remove paths.)

 

The FILE dropdown menu of the file-editor allows you three choices:

 

   1. File->Save Allows you to save the current list as clean_batch_list.txt. File->Save defines the set of files to be cleaned.

 

   2. File->Save As Allows you to save the current list under a different name. NOTE: A list saved in this way will not be cleaned in this run, but can be used as input to a future run of IRSCLEAN (either by picking it in the dialog window, or entering it using the command line keyword filesToClean).

 

   3. File->Clean Files Now Exits the file editing window and cleans the set of files in the currently saved .txt file clean_batch_list.txt.

 

NOTE: Each session of IRSCLEAN will recreate clean_batch_list.txt, so if you wish to keep your list of files to clean for future sessions, remember to use

‘File->Save As’ to write a unique file.

7.3       Cleaning Files

After you have edited the list to your satisfaction and select File->Clean Files Now, the files in your list will be cleaned using your rogue mask and IRSCLEAN. The cleaned images will be stored in .fits files with the same name as the original images, but with _clean inserted before the .fits suffix. For example the clean image corresponding to 'sampledata/r13349376/ch3/bcd/SPITZER_S3_13349376_0012_0000_8_bcd.fits’ is stored in 'sampledata/r13349376/ch3/bcd/SPITZER_S3_13349376_0012_0000_8_bcd_clean.fits’. A HISTORY keyword is added to the FITS header in the clean file, specifying the date and time that IRSCLEAN was run on the file.

 

NOTE: If a file in your list does not exist or is not a fits file, you will see a message to that effect. No action will be taken for that entry. Also, if a file is not from the same IRS array as the rogue mask, it will not be cleaned.

7.4       "Cleaning" Uncertainty and Mask Files

 

Table 7.1 Uncertainty and Mask file names

 

For every data file in your list, IRSCLEAN will automatically look for a corresponding uncertainty and bmask file and produce “clean” versions of those files as well. The relationship between the name of a data file and that of its uncertainty and bmask files is shown in Table 7.1. To be processed, uncertainty and bmask files need to be in the same directory as the processed image. Otherwise no action is taken.

 

Just as for the data files, the “cleaned” uncertainty and bmask images will be stored in .fits files with the same name as the original files, but with _clean inserted before the .fits suffix. For example, the cleaned version of 'sampledata/r13349376/ch3/bcd/SPITZER_S3_13349376_0012_0000_8_func.fits’ is stored in 'sampledata/r13349376/ch3/bcd/SPITZER_S3_13349376_0012_0000_8_func_clean.fits’.  A HISTORY keyword is added to the FITS header in the clean file, specifying the date and time that the file was processed.

 

If found, the uncertainty and bmask files are altered by IRSCLEAN as follows:

 

• The uncertainty value in cleaned pixels is interpolated from neighboring pixels using the same algorithm as for data. This ensures that the pixel has minimal impact on the computation of the uncertainty in the corresponding wavelength bin of the extracted spectrum. In earlier versions, uncertainties of cleaned pixels were set to NaN (not a number). This had the inadvertent effect of eradicating (setting to NaN) the error estimate for the entire wavelength bin of a cleaned pixel.
• The bmask value in cleaned pixels is set to 1 (one). In other words, we turn on the zeroth bit and turn off any other bits that may have been set in the bmask by the pipeline. Only pixels cleaned using IRSCLEAN will have the zeroth bit set in the bmask. Setting the bmask value of cleaned pixels to 1 also ensures that cleaned pixels are always included in spectral extraction in SPICE, since any fatal masking of rogue pixels by the BCD pipeline will be cleared.

• 
• 
• 
• 
•