The binaries are stored in $SIRTF_BIN ($SOS_GeRT/bin).
3.1 CVTI2R4G
Ge-version of CVTI2R4 (which avoids FORTRAN specific compiler issues). Converts integers to real values and makes initial bmask and dmask files. Also marks missing data and saturation in the dmask. The bmask is set to 1 for stim DCEs (STMFL_70, STMFL160 > 0) and 0 for non-stim DCEs (STMFL_70, STMFL160 = 0.0).
IMPORTANT PARAMETERS:
&CVTIN
DataHi = 0,
DataLo = 1,
SatHi = 65500,
SatLo = 10,
StimHi = 4,
StimLo = 4,
DataHi: number of reads to ignore at end of non-stim DCE
DataLo: number of reads to ignore at start of non-stim DCE
SatHi: DN value for high saturation (set dmask to ignore for higher DN's)
SatLo: DN value for low saturation (set dmask to ignore for lower DN's)
StimHi: number of reads to ignore at end of stim period DCE. The last 4 frames are taken
after the stim is turned off.
DataLo: number of reads to ignore at start of stim DCE (ignore stim warm-up period).
3.2 SATURATION
Sets dmask samples with DN values above the values given in the MIPS*_SAT.fits calibration file. Saturation can be set to different values for different pixels.
3.3 ELECNL
Applies electronic nonlinearity calibration to ramps as a function of DN using MIPS*_ENL.fits calibration file. Uses a cubic spline to interpolate between table values.
3.4 RESET
Checks for resets in the data ramps via header keywords. The reset will occur after the frame given by RSTP160/(2^COADD) for 160 micron and RSTP_70/(2^COADD) for 70 micron.
IMPORTANT PARAMETERS
&RESETIN
number_pixels_to_ignor = 4,
number_pixels_to_ignor: number of reads to ignore after the reset.
3.5 RADHIT
Performs radhit detection using a Bayesian probability technique that checks for ramp discontinuities. A ramp jump above the threshold is declared as a radhit in the dmask and the ramp segments on each side of the jump are checked again for other radhits. The process continues until no more radhits are detected or until the maximum number of hits are detected. The module uses input readnoise and radhit statistics. It is possible to provide an input readnoise calibration file, which takes priority over the Readnoise namelist parameter (to account for possible pixel-to-pixel readnoise variations, e.g., MIPS*_rnoise.fits). One can tune up RADHIT separately for stim data (RADHITSTIM block) and non-stim data (RADHIT block).
IMPORTANT PARAMETERS
&RADHIT
FITS_In_Readnoise = ./cal/MIPS70_rnoise.fits,
Readnoise = 100,
NominalRHMag = 5,
RHPriorProb = 0.01,
DeclThresh = 0.99,
MaxNumHits = 16,
NumSamplesMax = 40,
Gain = 7.1,
NumDeclareBadAfterRH = 4,
ThreshDeclareBadAfterRH = 10000,
Readnoise: input readnoise in electrons
NominalRHMag: Typical RH mag in terms of x Readnoise (e.g. 5*readnoise)
RHPriorProb: Prior probability for a sample to be hit by a RH.
DeclThresh: Probability threshold for declaration of RH.
MaxNumHits: Maximum number of RHs in ramp before stop searching for RHs.
NumSamplesMax: Maximum number of samples to use in calculation. Longer ramps (e.g., 10sec = 80) are broken into two separate ramps to search for radhits. This is done for speed consideration, since RADHIT inverts a probability matrix the processing goes with ~n^2 instead of n. Breaking into 40 samples does not affect the module's ability to find radhits.
Gain: Conversion between DN to electrons, 7.1e-/DN.
NumDeclareBadAfterRH: Number of reads to ignore after a strong radhit with magnitude larger than ThreshDeclareBadAfterRH.
ThreshDeclareBadAfterRH: DN threshold for declaring reads bad after RH.
3.6 SLOPE
Slope performs a linear fit to the segments defined in the dmask. Requires at least 4 samples for a slope estimation. Standard linear regression is done and the scatter of the fit provides the uncertainty for the segment.
IMPORTANT PARAMETERS
&SLOPE
Min_Num_Samples = 4,
Min_Num_Samples: Min number of samples needed for a slope calculation.
If user picks Min_Num_Samples < 4, one can fit for fewer samples, but RADHIT requires 4 samples to check the end-points properly.
3.7 FUSION
Does a weighted average of the slopes from the segments to derive final slope. For example, for two slope segments: s1+/-u1 and s2+/-u2, the final slope = wt1*s1 + wt2*s2 where wt1~1/u1^2, wt2~1/u2^2, and the slope uncertainty~([1/u1]^2 + [1/u2]^2)^-0.5.
IMPORTANT PARAMETERS
&FUSION
Negative_Rejection = 3,
Outlier_Rejection = 20,
Negative_Rejection: threshold "sigma" level at which negative slopes are ignored. If slope measurement is < -1*Negative_Rejection, then this slope segment is not included in the slope calculation.
Outlier_Rejection: threshold "sigma" level required for including segments in slope calculation. Some strong radhits can significantly change the responsivity of a detector such that the remaining part of the ramp should be ignored. If the slope measurement after the radhit is more than Outlier_Rejection times sigma different than the measurement before the radhit, the segment after the radhit is ignored. In general, "sigma" for the FUSION module significantly underestimates the true uncertainties in the slopes, so a higher Outlier_Rejection parameter is needed than would otherwise be expected. One could be more aggressive in fusion rejection at the expense of throwing away data.
3.8 MASKWRITE
Copies information from the pmask and dmask to the bmask. Information from the dmask to bmask are copied in cases where data within in ramp are missing, saturated, or contains a radhit.
ged_SAMPSMISSING -> geb_SAMPSMISSING
ged_SATURATEHI -> geb_SATURATED
ged_RADHIT -> geb_RADHIT
gep_BADHALF -> geb_BADPIX
gep_BADPIX -> geb_BADPIX
gep_NOISY -> geb_NOISY
where ged =dmask, gep=pmask, and geb=bmask (see §3.15).
3.9 STACKLAYER
This program constructs a FITScube from single layers. Missing layers have the jam (e.g., NaN) value inserted. An index FITS file may be constructed from keyword values. (e.g. SCLK_OBS.fits). Only short, unsigned short (BITPIX = 16, BZERO=32768), and single-precision data types are supported.
USAGE
stacklayer -i <prototype layer to specify plane dimensions and type>
-k <keyword to create index file>, e.g., SCLK_OBS
-o <output fits cube>
-l <number of layers in output cube>
-j <jam value when nonexistent file>, e.g. NaN
-d (prints debug statements)
-m <list of conforming single layer file names>
-v (verbose output)
An input file is used to set the Naxis1 and Naxis2 dimensions. The "-l" parameter gives the Naxis3 dimension for the output cube. The "-j" option is used to fill NaNs for missing input current images and uncertainty images and "-j 0x4000" is used for missing bmasks.
3.10 INTERP
This module makes the stim response function by interpolating between stim-minus-background measurements.
IMPORTANT PARAMETERS
StimVariability = 0.05,
Comment = 'Method options: S = spline (global fit)',
Comment = 'X for weighted linear least squares',
Comment = 'P = piecwise linear (connect the dots)',
Comment = 'L = least squares (Order) polynomial fit (do not use L yet)',
Method = S,
IntegralWeight = 0,
NBracket = 2,
Power = 1,
Comment = "Flag the pixels for X seconds after stim with the mask",
FlagAfterStimTime = 11,
FlagAfterStimMask = 32,
Comment = " mask bit to mark pixels with extrapolated stims ",
ExtrapolatedMask = 64,
StimVariability: %error associated with individual stim measurements. A 5% error gives reasonable errors for the final BCDs/mosaics.
Method: interpolation method. Spline (S) is used online. "P" simply linearly interpolates between measurements. Only the "S" and "P" methods have been validated. Additional options are available, but have yet been fully tested. "X" is a weighted linear fit that uses NBracket stims on each side of the current DCE and weights as a function of time or DCE number (see details below). A general least-squares polynomial fit ("L") does not work yet. "X" may or may not work properly on your system (new un-validated feature for S14).
FlagAfterStimTime: time in seconds after stim to mask bit FlagAfterStimMask as data near stim warning. Can be used by MOPEX to ignore data near stim in coadd process.
FlagAfterStimMask: 32 (bit 5) bit masked for DCEs near a stim.
ExtrapolatedMask: 64 (bit 6) bit to mask DCES with extrapolated stim solutions.
Note that there are separate namelist blocks depending on the observing mode, such that it is possible to tune up processing based on data types. The modes are determined via the header keywords EXPTYPE and FOVID/APERTURE. The different "modes" are:
FINE: fine-scale SCI photometry (only applicable for 70um) Science FINE set for EXPTYPE = pht + FOVID > 117
SED: SED mode SCI observations (only applicable for 70um), EXPTYPE = sed
TP: TPM mode SCI observations, EXPTYPE = tpm
DARK: For DARK pipeline processing, EXPTYPE = d2a
EXPTYPES for different IC's are checked via hash table from w_mips_cdfblock.pl to choose the proper block.
The interpolation is performed using a table where the y value is stim-background. The background for a stim is the immediate preceding DCE where it exists, or the immediate following DCE if the stim DCE is the first in the FITScube. The uncertainty of the y value in the table is the root-sum-square of the uncertainty in the background, the uncertainty in the stim and a stim-to-stim variation calculated as StimVariability * (stim - bkg). The x value in the table is the time of the stim. Stims may be missing, and a missing stim is simply not entered into the table. In cases of double stims (e.g., when stacking multiple scan legs together), the 2nd stim frame is ignored.
For the SPLINE technique, the independent variable is SCLK_OBS for the DCE. Thus, stims may be missing and the smoothness of the spline fit is relied upon to provide a reasonable interpolation through missing stims. If an interpolation is needed outside the spline table (i.e. before the first stim or after the last stim) , the spline interpolation routine is designed to perform a linear extrapolation.
For the least-squares technique (method = "X"), the chi squared linear fit routine from Numerical Recipes is used. Several tunable parameters are available to control the fit method:
NBracket: the number of stims on each side of the current DCE in fit, note: 0 means use all stims;
Power: exponent of the time difference (k);
IntegralWeight: = 0 for using the actual times differences as weights, or = 1 for using the ceiling of the time differences (weight as function of the number of Stim DCEs from the current DCE).
Weights are determined by the distance in time between the DCE time and stim time.
Two possibilities for weights calculations are:
If IntegralWeight = 0 then:
Equation 3.1
and if IntegralWeight = 1 then:
Equation 3.2
where,
T = time of DCE for stim interpolation
Tstim = time of stimflash itself
Tdelt = interval between stims
Tdelt is calculated as the minimum delta-T between stims in the AOR. Tdelt is irrelevant to the calculation of the least squares coefficients if IntegralWeight = 0 is chosen, but will allow the least squares weight calculation to be performed less often for the integral difference algorithm (IntegralWeight = 1). For example, with integral differences between T1 and T2:
time T0 T1 T2 T3
weight 2**(-k) 1**(-k) 1**(-k) 2**(-k)
Note that integral weights may not be useful for photometry AORs because the delta-t between stims varies with the dither patterns.
The number of stims used to calculate the least squares coefficients is 2*NBracket, unless NBracket = 0 then all stims in the AOR are used to calculate the least squares coefficients.
3.11 SLOPECAL
The SLOPECAL module carries out the calibration and filtering for the MIPS-Ge pipelines. The processing done by the module is controlled by the inputs such that operations are skipped when no inputs are provided.
If a DARK, IC, and FC are given as input, the pipeline performs the following science calibration:
Equation 3.3
where:
I(t) = unfiltered BCD product, nofiltercube.fits
U(t) = input slope image, curcube.fits
S*R(t) = stim response function from INTERP, interpstim.fits
DARK = dark calibration file from cal/MIPS*_DARK.fits
IC = IC calibration file from cal/MIPS*_ILCORR*.fits
FC = flux conversion factor which is given via a cal file.
If no FC and DARK are given as input, then I(t) = U(t)/S*R(t). If no FC and IC are given, then I(t) = [U(t)/S*R(t) - DARK]. If no FC, IC, and DARK are given, then I(t) = U(t), which is used for 2nd pass filtering.
At 70 micron and 160 micron a high-pass median time filter is done on a pixel basis, and an additional column filter is done for 70 micron. The filtered fbcd products form the calcube.fits file.
As with INTERP, there are separate namelist blocks depending on the observing mode, such that it is possible to tune up processing based on data types. The modes are determined via the header keywords EXPTYPE and FOVID/APERTURE. The different "modes" are:
FINE: fine-scale SCI photometry (only applicable for 70 micron). Science FINE set for EXPTYPE = pht + FOVID > 117
SED: SED mode SCI observations (only applicable for 70 micron), EXPTYPE = sed
TPM: TPM mode SCI observations, EXPTYPE = tpm
DARK: For DARK pipeline processing, EXPTYPE = d2a)
EXPTYPES for different IC's are checked via hash table from w_mips_cdfblock.pl to choose the proper block.
IMPORTANT PARAMETERS
JanskyScaleFile = cal/MIPS70_fluxconv.tbl,
BUNIT = 'MJy/sr',
median_count = 16,
median_variance_option = 1,
Comment = 'nonzero means column filter is on',
colfilt = 1,
colfiltfirst = 1,
JanskyScaleFile: Calibration file for conversion between instrument units to MJy/sr)
median_count: high-pass time filter width in DCEs.
colfilt: only used for 70um. colfilt = 1 means apply column filter
colfiltfirst: only used for 70 um. colfiltfirst = 1 means apply column filter before high-pass filter.
Users may want to test the quality of the filtering corrections on their own data by switching the order of the filters (colfiltfirst = 1/0) and modifying the high-pass filter width (median_count = 10 -- 50 DCEs).
The high-pass median time filter cannot be run stand-alone (requires call to SLOPECAL). The median filter subtracts the median value of the surrounding DCEs on a pixel-by-pixel basis. If the namelist parameter median_count is nonzero, then up to median_count of samples will be extracted from the FITScube by looking for non-stim and non-NaN values in the following order: -1, +1, -2, +2, -3, +3, -4 +4 ... until median_count values have been found. Stims are not affected. Pixels that are NaN's are not affected. The selection process will not go outside the boundaries of the FITScube, which means the median buffer for the last non-stim pixel is the median_count non-stim pixels preceding it.
3.12 COLMN_FLTR
Applies column filter for MIPS-70 fbcds. Normally COLMN_FLTR is used as a function within the SLOPECAL module, but it can be run offline stand-alone.
INPUT
a) File: 3-D fits file.
b) File: corresponding bmask.fits.
c) File: corresponding uncertainties fits file.
OUTPUT
a) File: column_filtr.fits (input image after subtracting the column medians (double)).
b) File: column_filtr_uncertainties.fits (file with uncertainties (double)).
DISCUSSION
Column filter should subtract the median of the values of the good pixels in the column for each column for every MIPS-70 filtered-bcd. Bad readout region on good side of the array is ignored by using the information in the bmask.
3.13 GETLAYER
This program extracts a single plane from a fits cube and writes it as a simple fits file. The keywords from the input file are copied, with modifications to the naxis keyword. Only short, unsigned short (BITPIX = 16, BZERO = 32768), and single-precision data types are supported.
USAGE
getlayer -i <input fits cube>
-k <input simple fits file with keywords to copy>
-o <output simple fits file>
-z <bzero>
-h <HDU>
-l <layer>
-d (prints debug statements)
-m <multifile list file> overrides -k and -o options
-v (verbose output)
The header from fits file "-k" is attached to the pixel data for a layer of a data cube "-i". Loop through the cube with index "-l". GETLAYER and STACKLAYER are used by the GeRT to move data in and out of cubes. The uncalibrated slope images need to be stacked into a data cube for calibration and filtering (software was designed this way). After calibration, the cube is split apart into individual BCD products and headers with pointing from the slope images are re-attached.
3.14 ge_mask_pointsource
Copy of the mask_pointsource script from MOPEX. Given a list of point source coordinates either in x,y (then FIF is required), or in RA,Dec decimal degrees, the module masks the user specified bit for the pixels within the user specified circle or box centered on the point sources. The number of point sources projected onto each mask is recorded in the header with the keyword NPSMASKD.
There are two usage modes specified by the Input_Use keyword. If Input_Use = 'copy_input_data', then the output masks are the result of "OR"-ing the input masks and the "point source" bit. If Input_Use = 'copy_input_header', then the output masks are created from scratch. In this case the data type of the output masks is a single byte (8 bits, BITPIX = 8).
For either usage mode a list of input images is required for the header information, such as image sizes and pointing. If Input_Use = 'copy_input_header', the input images don't have to be masks. One can use any images, e.g. BCDs, with the relevant header information.
Output file names can be specified either in the OutputListFileName or created from the input files file names. In the latter case an Output_Prefix should be specified. The output file names are created by replacing the path in the input file names with the Output_Prefix. If Output_Prefix contains subdirectories names, the subdirectories have to have been created. In order to overwrite the input masks with the output masks set OutputListFileName = ImageListName. For S14.0 users can use the MOPEX detect image for masking instead of a source list.
See the GeRT cleanup scripts to see how to call binary from the command line.
3.15 mips_ge_mask.h
The MIPS-Ge mask definitions are summarized below (mips_ge_mask.h is an include file used in the binaries). The PMASK is a static cal file, while the bmask is made from the processing for each BCD.
S14 PMASK:
Bit Comments
2 Noisy pixel
3 Bad electronic nonlinearity
8 Bad half of 70um array
14 Bad pixel
S14 BMASK:
Bit Comments
0 Stimflash DCE
1
2
3 Saturated sample(s) found in raw data
4
5 Data near stimflash warning
6 Stim extrapolation warning
7
8 Raw DCE has missing data
9 Radhit(s) found in raw data
10 Uncertainty Status
11 Bad pixel as defined by the pmask
12 Slope calibration failed
13 Slope calculation failed
14 Missing layer, NaN and/or bad data
Details with hex bit values.
Naming conventions:
gep prefix - pmask file definition
ged prefix - dmask file definition
geb prefix - bmask file definition
gec prefix - cmask file definition
Very often bits have identical semantics in different files because bits get copied from one file to another. It is highly desirable to have spelling identical (as far as it can be) in this case. Thus the prefix is invented.
PMASK (Static pixel mask file)
#define gep_NOISY 0x0004
#define gep_BADHALF 0x0100
#define gep_BADPIX 0x4000
DMASK
Ramp status bits for each pixel contained in a FITScube (thirddimension is sample number in ramp).Produced by sloper pipeline
#define ged_DIDRHDET 0x0004
#define ged_SATURATEHI 0x0008
#define ged_SATURATELO 0x0010
#define ged_SEGMENT 0x0100
#define ged_RADHIT 0x0200
#define ged_BADSAMP 0x0800
#define ged_RHBADSAMP 0x2000
#define ged_SAMPSMISSING 0x4000
BMASK
Status bits for each pixel contained in an image plane. Produced by sloper pipeline, updated by caler pipeline.
#define geb_STIMDCE 0x0001
/* unused 0x0002 */
#define geb_NOISY 0x0004
#define geb_SATURATED 0x0008
#define geb_USER4 0x0010
#define geb_USER5 0x0020
#define geb_STIMEXTRAP 0x0040
/* unused 0x0080 */
#define geb_SAMPSMISSING 0x0100
#define geb_RADHIT 0x0200
#define geb_UNCERTSTATUS 0x0400
#define geb_BADPIX 0x0800
#define geb_CALERFAIL 0x1000
#define geb_SLOPERFAIL 0x2000
#define geb_MISSLAYER 0x4000
CMASK
Status bits for each pixel contained in an image plane. Produced by a calibration pipeline, supplied by caltrans to pipelines needing the cal file.
#define gec_NOISY 0x0004
#define gec_DARKFEW 0x0010
#define gec_ILCORRFEW 0x0020
#define gec_BADHALF 0x0100
#define gec_BADCAL 0x2000
#define gec_BADPIX 0x4000
3.16 IMHEADER
The binary imheader is a general tool used to read the headers of fits files.
USAGE
imheader file [files ...]
3.17 MIPSFLAT
Module used by IC pipelines to make IC product. For pixels not identified as bad in the mask files, mipsflat takes a trimmed-average per pixel of the dark subtracted data cube to derive the IC value for each pixel. The namelist block control mipsflat is "FLATFIELDIN" within the MIPS*_FLUXCAL_0.nl.
IMPORTANT PARAMETERS
cut_post_latent_dce = 0,
trim_prcnt = 48,
trim_prcnt_global = 50,
cut_post_latent_dce: the number of DCEs after the stim flash DCE to ignore.
trim_prcnt: the +/- trimmed percentage of DCEs from the wings of the distribution before. trim_prcnt = 50 gives a median.
trim_prcent_global: controls the normalization of the IC product (trim_prcnt_global = 50 gives median). Currently, the IC is normalized to the array median of the remaining good (non-bad and non-noisy pixels and non-NaN) pixels defined in the PMASK.
3.18 MIPSDARK
Module used by the DARK pipeline to make the DARK product. For pixels not identified as bad in the mask files, mipsdark takes a trimmed-median per pixel of the data cube after the stim response correction to derive the DARK value for each pixel. The namelist block control mipsdark is "MIPSDARKIN" within the MIPS*_FLUXCAL_0.nl.
IMPORTANT PARAMETERS
Cutoff_Outlrs = 2.5,
Outlrs_Percent = 5.0,
Skip_Post_DCE = 0,
Nan_Percent = 10.0,
Cutoff_Outlr: the sigma value about the median that is clipped before calculating the median per pixel.
Outlrs_Percent: the percentage threshold for the number outlier values per pixel before flagging in the output CMASK file.
Skip_Post_DCE: the number of DCEs after the stim flash DCE to ignore.
Nan_Percent: the NaN percentage threshold for the number NaN values per pixel before flagging in the output CMASK file.
