The Customizable User Pipeline for IRS Data (CUPID) allows users to run the Spitzer IRS Pipelines to re-create Basic Calibrated data from the archived raw files. Users may wish to do this in order to apply a different flatfield or dark correction to their data.
If CUPID is run in the default mode, it will produce output results very similar to the directories and files in the Spitzer Heritage Archive. Some of the ancillary files created in the bcd/ directory may differ in minor ways as detailed in Section 1.2.
1.1 Running CUPID
CUPID is run on the command line with the following syntax:
unix% cupid <parameters> <options>
For a list of available parameters and further details about running the “cupid” command, either see Chapter 2, or run the “cupid” command with no parameters. For a list of available options, see §1.4.
In the nominal case, CUPID runs through the following steps: first the BCD pipeline produces bcd images, next the COADD pipeline produces coadd images and then the BKSUB pipeline produces background subtracted images. After this, there are three pipelines, BCDX, COADDX, and BKSUBX, which do spectral extractions on the images.
For a list of all pipelines and the steps in each, see Chapter 3. For a list of all modules and the parameters for each, see: Appendix C: The Modules.
You could run the 7 standard IRS pipelines on an AOR by doing the following:
Use the Spitzer Heritage Archive to download the desired AOR. Only the "raw" data is needed if you have the full CUPID package. If you have the “NOCAL” version, you will have to download both the “raw” and the “calibration” files from the Spitzer Heritage Archive. (An AOR often contains 4 "processing directories", ch0/, ch1/, ch2/, and ch3/.) CUPID will attempt to create directories called bcd/ and/or pbcd/ (depending on which pipeline you run) alongside the downloaded raw/ directory. Therefore, make sure that you do not have existing directories called bcd/ and/or pbcd/ alongside your raw/ directory. If you have downloaded bcd and/or pbcd files from the Spitzer Heritage Archive, you will have to change the names of the default directories they are written into.
Run cupid with a command similar to the following, but pointed at your downloaded AOR:
unix% cupid /data/r1234567/ch*
1.2 Processing Directories
All of the pipelines are run in “processing directories”. There can be four processing directories for each AOR, one per channel (ch0/ ch1/ ch2/ ch3/). The directories output by CUPID will look very similar to those downloaded from the Spitzer Heritage Archive. Each processing directory will contain the following subdirectories:
bcd/ contains data files created by the single DCE pipelines. (BCD and BCDX) It also contains a .log file for each pipeline run. See "BCD Pipeline Description" document for details.
pbcd/ contains data files created by the multi-DCE pipelines. (COADD, COADDX, BKSUB, BKSUBX). It also contains a .log file for each pipeline run.
cal/ contains all calibration files used.
cdf/ contains all "control data files" (namelists) used.
1. The *.pline files in the Archive are the pipeline log files. These are named *.log in CUPID, one for each pipeline.
2. The *full.fits in the Archive have fully processed pixels but no final header updates. These are named *flatap.fits in CUPID.
3. The *spec2.tbl in the Archive is the extraction from the non-straycross processed files. These may be obtained by using the *spect.tbl files in CUPID run with the SKIP.STRAYCROSS option.
1.3 Status and Log Files
Every step run in CUPID returns a status. If a step returns a >=64 then processing is halted immediately. The meaning of status levels is as follows:
0 = completely nominal
1 = processing skipped because file is wrong type (not a problem)
2 - 31 = something happened out of the ordinary, but no problem
32 - 63 = something happened that may be a problem, processing continues
64 - 255 = something happened that is a problem, processing terminated
There is a log file for each pipeline run output subdirectory. The location of the log file is shown as the first line while running a pipeline. The log file contains pipeline run information and any error messages.
In addition, there is a STATUS file in the output subdirectory, which summarizes the status of the most recent run of each pipeline.
1.4 Options
CUPID can be called on the command line with a number of options, listed below. Note that the “SET” option uses module names but other options use STEP names. Also note, these options apply to the BCD pipeline unless they are prefixed with "X/", in which case they apply to all of the extraction pipelines (BKSBX, COADDX and BCDX).
SET (override parameter value):
This is used to set the value for a parameter when particular module is called. Each override specifier has the form:
SET.<module>-<parameter_flag>=<value>.
For example, a new flat file could be specified with:
SET.flatap-f=/urs/joe/data/newflat_1.fits
Or the following to specify 100.0 as the estimated readnoise when creating the uncertainty file:
SET.snest_irs-readnoise=100.0
START (start processing at a specified step):
This will cause the first step(s) to be skipped over. The specified step where processing will have the input files just as if the preceding steps had been run. This means that the pipeline must have been run from the first step at least once. For example, to begin processing at the CUBESUB step (using inputs from a previously run DARKBASE step), specify the following:
START.CUBESUB
STOP (stop processing after a specified step):
This will cause the the pipeline to stop processing after running the specified step. For example, to stop processing after the FLATAP step, specify the following:
STOP.FLATAP
SKIP (skip a step):
This will cause the the pipeline to skip the specified step. The following step will take its inputs from the preceding step. (This is true even if a previous pipeline run has produced data files for the skipped step.) For example, to skip the STRAYCROSS step, specify the following:
SKIP.STRAYCROSS
(In this case the following FLATAP step will use inputs from the DROOPRES step.)
DO.AFTER (move a step):
This will cause the the pipeline to execute the first specified step after the second specified step. The first specified step must precede the second specified step in the nominal processing order. For example, to swap the order of CUBESUB and DARKBASE, specify the following:
DO.DARKBASE.AFTER.CUBESUB
The full command to produce "unstraycross" BCDs if the pipeline has already been run would be the following. Note that it starts with the FLATAP step to avoid needlessly re-running the first few steps.
The "SKIP.STRAYCROSS" option here will cause FLATAP to get its input from the DRROPRES output files rather than from the STRAYCROSS ones.
1.5 Verbosity
The default level of output should be sufficient in most cases, but there may be cases where more or less output is desired. There are environment variables to control the amount of information output.
The default value of CUPID_SHOW is 2. It can be set to a lower value to reduce the clutter on the users terminal. As each pipeline runs, the default is to display the following:
the output log file location;
step-by-step status (if CUPID_SHOW >= 1);
final status.
After all pipelines have run, a summary is displayed at the end (if CUPID_SHOW >= 2).
1.5.2 Log Files (CUPID_LOG environment variable)
The default value of CUPID_LOG is 2. The optional contents are as follows:
the actual command line call used to process a step (if CUPID_LOG >= 1);
values of FITS words read from file (if CUPID_LOG >= 2);
information about fetching cal and cdf files (if CUPID_LOG >= 2);
values explicitly set environment variables (if CUPID_LOG >= 2);
values of defaulted environment variables (if CUPID_LOG >= 3);
timing summary per step (if CUPID_LOG >= 3).
1.6 Fetching cal or cdf Files
Whenever a module needs a calibration file (cal) or control data file (cdf), CUPID must decide which file to use. Here is the logic used:
If the file is already there in the local cal/ or cdf/ directory, it is used and no actual "fetch" is required. A file is considered "there" if:
a. it has the correct root file name, and
b. it has the correct channel number (b0_ b1_ b2_ b3_), and
c. if it is a dark, it must have the correct exposure time designator (r6_ r14_ etc.)
The SCLK range in the file name is not considered.
If the file is not in the local directory, if will be "fetched" (copied) from the CAL or CDF archive directory. The file fetched will be the one that is qualified as follows:
a. it has the correct root file name, and
b. it has the correct channel number (b0_ b1_ b2_ b3_), and
c. if it is a dark, it must have the correct exposure time designator (r6_ r14_ etc.);
d. it has a SCLK range that covers the data being processed.
If more than one file qualifies, the one with the smallest SCLK range will be used.
1.6.1 Alternate cal or cdf Archives
If you have a directory with an alternate cal/ or cdf/ archive, it can be used by setting the CUPID_CAL_ARCHIVE or CUPID_CDF_ARCHIVE environment variables.
1.7 CUPID Directory Structure
The CUPID/ directory contains the following files and subdirectories:
cupid the main CUPID command script
HELP user information in plain text format
HMTL user information in webpage format (point your browser here)
VERSION information about this specific version of CUPID
wrappers/ pipeline "wrapper" scripts (called by cupid command, not directly by users)
cal/ archive of calibration files
cdf/ archive of "control data files" (primarily namelists)
bin/ executable modules
include/ "include" files used by executable modules
lib/ "lib" (subroutines) files used by executable modules
1.8 Wrapper Scripts
The actual pipeline processing logic is in wrapper scripts in the CUPID wrappers/ subdirectory. These wrappers specify the pipeline "steps" and module calls for each pipeline. Parameters passed to modules are specified as follows:
IN input data files
· may not override
· are prepended with prefix (e.g. SPITZER_12345678_S3_0003_0001_)
· may use wrapper variable (e.g. <FLUX>) as filename
· may also set wrapper variable (e.g. <FLUX:ALT>) to the filename
OUT output data files
· may not override
· are prepended with prefix (e.g. SPITZER_12345678_S3_0003_0001_)
· may set wrapper variable (e.g. <FLUX>) to the filename
UPD updated data files (data files which are input AND updated)
· may not override
· are prepended with prefix (e.g. SPITZER_12345678_S3_0003_0001_)
CAL calibration files (input, in cal/ subdirectory)
· may override
· are in the cal/ subdirectory if not overridden
· fetched from CUPID cal archive if not there already
CDF control data files (namelists; input, in cdf/ subdirectory)
· may override
· are in the cdf/ subdirectory if not overridden
· fetched from CUPID cdf archive if not there already
