This section describes changes to the IRS pipeline that accompany changes to the overall Spitzer pipeline. Roughly every six months, the pipeline number (the "S-number") for all new Spitzer data in the archive is increased. New data will appear in the archive with a different S-number than old data. Parallel to these changes, the IRS pipeline may be improved and those improvements are described below. However, old IRS data will not be reprocessed with new pipeline versions unless the are substantial changes in the data quality or the calibration files. Users that have already downloaded data do not need to download it again when new pipeline is released. When pipeline changes result in substantial improvements to the IRS data, all past campaigns will be reprocessed with the newest pipeline.
S18.18 (Release Date: January 2011)
1. Outlier rejection logic improved for ramp-fitting in SLOPEFINDER. This helps to reduce noise in rare cases where there is one bad image plane.
2. Improved and reduced pipeline uncertainty estimates. The new pipeline module SNEST_IRS2 revises the uncertainty estimate after droop correction. The biggest effect is seen for faint sources in SL, where droop is enhanced by flux from the Peakup Arrays.
3. Uniform DCE weighting in coadd images. Previously, DCEs were flux-weighted in coadds. This caused spurious results, particularly noticeable in the SH module, including greatly enhanced noise in a fraction of AORs where spurious negative values might give rise to small or negative weights. For more details, see http://irsa.ipac.caltech.edu/data/SPITZER/docs/irs/features/#highres_coad_bug.
4. Slightly revised LL1 wavsamp file to better follow stellar trace. A small sub-pixel shift at the blue end of LL1 results in a better fit to the position of the profile on the LL array and more recovered flux.
5. Finer sampling of the object profile to more accurately determine the ridge location for extraction (Ncuts= 1000, 200, 1000, 200 for SL,SH,LL,LH), and to match the SPICE profile sampling.
6. Rederived all flux calibration files, using data through the last IRS campaign.
7. SL flux calibration split to account for IRS Campaign 33 frametable change. This results in slightly higher (<2%) SL1 fluxes and better low-resolution order matching for pre-33 data.
8. SH and LH fluxes reduced by 2% at all wavelengths to match low-res flux calibration, based on standard star HR 7341.
9. PUI zeropoints and aperture corrections updated using the best available observations of standard stars from all IRS campaigns.
10. Blue and red peakup cutouts are now produced for all SL calibration data. These cutouts are calibrated and processed using the standard peakup imaging pipeline.
11. Peak-Up Acquisition images are now processed through a separate pipeline which calibrates these images and produces standard data products.
S18.7 (Release Date: May 2009)
1. The wavelength calibration files have been updated for S18.7 processed data. The primary change is to the wavsamp.tbl files for LL, SH, and LH, implementing improved wavelength calibration solutions.
2. The CDF files have also been updated for S18.7. SL and LL spectra are now divided by the height of the wavsamp extraction pseudo-rectangles, resulting in better matching orders in extract.tbl files.
3. The S18.7 flux calibration files (fluxcon.tbl, aploss_fluxcon.tbl) have been updated to account for the new wavelength calibration and CDF files described above. The SL, LL, SH, and post24 LH flux calibration are also refined by including additional calibration data from recent IRS campaigns.
4. The SH and LH flat-fields have been updated to account for the new wavelength calibration.
5. The bug causing duplicate and missing nods for background subtracted SL and LL staring mode data has been fixed. See IRS Important Note from 30 Jan 09.
6. The PU imaging calibration zeropoints are refined by including additional calibration data from recent IRS campaigns.
7. PU image cutouts have been created from SL observations.
9.5.1.1 S18.1 (Release Date: Aug 2008)
1. The file names of background-subtracted low resolution products (bksub) now include correct EXPID labels. Formerly, these labels incorrectly referred to the first EXPID in the AOR, instead of the actual EXPID.
2. Three new keywords have been added to PUI mosaics (post-BCD data): PA, PXSCAL1, and PXSCAL2. These keywords replace the CROTA2, CDELT1, and CDELT2 keywords in those data.
9.5.1.2 S18.0.1 (Release Date: Jun 2008)
1. Spectral table files are now provided also in FITS format
9.5.1.3 S17.2.0 (Release Date: Apr 2008)
1. The LH spectral extraction algorithm and corresponding flux calibration have been updated. The flux values (in e/sec) for LH delivered by the Extract module (Regular or Optimal) are now divided by the height (measured along the spectrum in pixels) of the extraction window, as set by the wavsamp wavelength calibration file. This change improves the order matching in *_extract.tbl spectra, but does not have much effect on the final, flux calibrated, *_spect.tbl spectra delivered by Tune. These updates do not pertain to any of the other modules.
9.5.1.4 S17.0.0 (Release Date: Jan 2008)
1. Errors due to readnoise and Poisson noise are now incorporated in the pipeline uncertainties, and are propagated during slope estimation. Uncertainties are now propagated during background subtraction (bksub).
2. A new slope estimation module was developed. Slopes (fluxes) are now computed via linear fits to entire ramp, as opposed to individual slope fitting to consecutive samples.
3. Better treatment of ramp slopes after a radhit. The S16 pipeline discards all samples of a ramp following a radhit. This leads to improvement in the signal-to-noise for low illumination levels, but to a slight (10%) increase in noise in high resolution data, in the presence of many radhits (several hundred radhits per array). For S17, the ramp following a radhit is discarded only if the slope is <500 e-/s (before the radhit).
4. Darks are now campaign-dependent for LL, built from data taken in a symmetric moving window 5 campaigns wide. This improves the cosmetic appearance of non-background subtracted data but results in negligible differences for sky subtracted data.
5. The LL bias and temperature were changed in campaign 45 to reduce the number of rogue pixels. The decrease in gain resulting from the change is more than compensated by the reduction in rogue pixels, leading to a significant improvement in overall S/N (10-30%). New darks, nonlinearity coefficients, flats, and flux calibrations were derived for the new settings and used to process the S17 post-44 campaign LL data. Additional improvement was made in the post-44 flat-field spatial correction, leading to better nod and order matching.
6. The PU imaging mode has been recalibrated in order to reduce the zeropoint uncertainty, which is now 2%. The zeropoint is now tied to an infinitely large aperture. More details are given in the IRS Instrument Handbook (http://irsa.ipac.caltech.edu/data/SPITZER/docs/irs/irsinstrumenthandbook/)
9.5.1.5 S16.1.0 (Release Date: May 2007)
1. During slope fitting in SL, SH, and LH, all samples are discarded following a radhit. This implementation is already active for LL, since S15. The reason for this change is that the slopes of faint ramps can change by factors of 30% or more following a radhit (w.r.t. before the radhit).
2. The radhit module now allows to skip the first sample of a ramp during its detection process. Therefore the module now avoids the spurious detection of radhits in the second sample of a ramp. These spurious detections were caused by the first sample being depressed with respect to the rest of the ramp slope, in modules such as SL. This implementation of skipping of the first sample does not apply to 4-sample ramps, because the radhit module needs a minimum of 4 samples to work.
3. All data taken from IRS campaign 40 onward (15 Apr 2007) will be processed with S16.1.0. Most previous data remain S15, with the exception of LH IRS campaigns 29 to 34 and campaign 36, as well as SL IRS campaigns 38 and 39. See the archive to Important Notes to Observers, note from 31 Jul 2007.
9.5.1.6 S16.0.0 (Release Date: Apr 2007)
No changes.
9.5.1.7 S15.3.0 (Release Date: Feb 2007)
All IRS data have been re-processed with S15.3.0 and are available for download.
Listed here are the most significant changes in this version.
1. To reflect all the pipeline changes, new flux calibration polynomials (fluxcons) were derived for all modules. Among other things, the new fluxcons corrected the 5% red tilt in LL2 present in S14 (see http://irsa.ipac.caltech.edu/data/SPITZER/docs/irs/features/, "Low-Res Nonrepeatability"). LL1 spectra now have slopes that are steeper (redder) by 3% over the spectral range, with respect to S14.
2. New flat-fields were built for all modules
a) The spatial behavior in all the flat-fields was improved. Observations in two different nods now match to better than 2%.
b) SH and LH flat-fields are now corrected using extended zodiacal light observations.
c) SL and LL interference fringes reduced to 1-2% level for sources well centered on the slit.
d) Bad regions in LH flat masked out to remove "bumps" at 20 microns.
3. Updated SH, SL, and LL wavsamp files
a) SH: The wavelength solution updated in all orders. The largest changes occurred at orders 19 and 20 (<0.5 pix). Spatial profiles changed in orders 11-15 only.
b) SL, LL: Updated spatial profile for all orders. The change is especially apparent in the LL2 ridge location. No wavelength changes were made, beyond very small shifts arising from the spatial profile change.
4. BCD products contain new header keywords VHELCORR and VLSRCORR to allow correction for the orbital velocity of Spitzer
The values of VHELCORR and VLSRCORR are to be added to velocities derived from IRS spectra, to obtain final velocities in the Heliocentric or LSR frames respectively. These keywords will be of interest only to observers measuring velocities with high precision from Short- High or Long-High spectra.
5. Revised uncertainty propagation
a) Suppressed propagation of the correlated component of uncertainty during residual droop correction.
b) During 2-D co-addition of BCDs (which produces the postBCD products *coa2d.fits and *bksub.fits), the output uncertainty (*c2unc.fits and *bkunc.fits) is now the root-sum-square of the input BCD uncertainties, divided by the number of co-added BCDs. Formerly (S14) the output uncertainty was the trim-standard deviation of the input BCDs.
c) The resulting uncertainties do not yet include all the systematics (photon noise per sample, readnoise, etc) associated with the processing.
6. Spectrum extraction heights and widths
a) SL and LL default extraction height changed from 1 pixel to native wavsamp height, to better follow spectrograph resolution.
b) SL3 default extraction aperture widened to 5.3333 pixels at 8.0 microns, to match SL2.
7. Updated SL, LL, and SH super-darks
New observations of the dark spot were obtained at times when the zodiacal emission was lowest. This change is most apparent for LL, which used to have a mixture of high and low zody data.
8. Radhit rejection in LL ramps
All samples of a ramp following a radhit are discarded during slope estimation in Long-Low data. This results in improved signal-to-noise in faint long-low data. Future pipelines will implement the rejection in other modules.
9. The calibration files for the Peak-Up Imaging mode have changed
a) Absolute calibration values and aperture corrections have been updated, which results in changes in measured photometric values of <15%.
b) New PRFs have been provided for APEX.
c) The bit masks have been updated to better interact with the mosaicker from MOPEX.
d) Fixed bug in error propagation. Users still need to remember the systematic zeropoint uncertainty for themselves.
All IRS data taken during or after IRS Campaign 30 (i.e., after the DATE_OBS keyword = 2006-04-16T00:00:00) have been processed with the S14.0.0 pipeline. Except for the special cases mentioned below, the pipeline is identical in every way to that of S13.2.0 and S13 versions of SPICE will work seamlessly with S14 data. (No changes have been made to fluxcon files or other SPICE related files for S14).
Special reprocessing of IRS campaigns back to IRS Campaign N2
Two targeted changes have been made to archived data for S14 which will lead to improvements in the quality of these data.
1. All IRS Long-Low (Channel 2) data, extending back to Campaign N2 in IOC, have been reprocessed through S14.0.0 to improve the linearity correction for saturating sources. The largest changes (15-20% in some cases) will be found for targets with flux densities well above 1 Jy, in regions of the spectrum where the ramps approach or exceed saturation. Targets with fluxes less than 1 Jy will, in general, be less affected unless long ramps were used leading to saturated/near saturated values. The improvements manifest themselves as smoother spectral continua in the 20-30 micron range, and reduced noise in this region. In all cases the apparent flux density in the affected region of the spectrum will go down, since the linearity correction has been reduced. Users should not be worried that they have mixed S13 and S14 data for targets involving observations with both SL and LL prior to Campaign 30. This is to be expected since the S13 and S14 pipelines are identical for SL (as they are for SH and LH).
2. All IRS Peak-up imaging AORs were reprocessed through S14.0.0 to ensure that improved 16 and 22 micron PRF files (Point-source Response Function) were correctly included with archived products. Previous versions of the pipeline had provided for the user a preliminary version of the PRF which could have led to problems with automated extraction of sources in MOPEX. The reprocessing and archive to these new PRF files ensures that the automated calibration, if run by the user, will lead to results consistent with those fluxes derived by manual extraction. Further refinements of the red PRF are underway and an even better version will be available in S15.
9.5.1.9 S13.2.0 (Release Date: Jan 2006)
All IRS data taken in Nominal Ops campaigns 1-27, and SV campaigns N2-R have been re-processed with S13 and are in [the process of being] archived. Check this page to see when your favorite campaigns have been archived and are available for download.
There are a number of significant changes to these data processed through the S13 pipeline. These are:
Wavelengths
* Wavelength calibration updated for all Short-Hi orders.
* Spatial centering of the wavelength solution was changed for most modules.
- Largest LH shift is 0.5 pixels for order 11.
- Largest SH shift is 0.3 pixels for order 20.
- 0.68 pixel shift of LL2.
- LL slit length was increased to 34 pixels.
* The sampling of the LL and SL "bonus" (order 3) wavelength solution was changed to match the gridding of order 1.
* LH wavelength samples were adjusted to ensure none fall completely off the detector array.
* The sign of the spatial offset in the b?_wavsamp_offset.fits files were changed for SL, LL and SH, to properly align with the sense of the PA_SLT keyword in the FITS headers.
* The scale of the LH b3_wavsamp_offset.fits file was updated to reflect the correct value of 4.5 arcseconds per pixel.
Flat Fields
Completely new (and improved) flat fields have been created for all modules, including a new flat field that applies specifically for LH data taken after the bias change to the detector at the beginning of IRS Campaign 25.
Flux Calibration
Significant refinements have been made to the Flux calibration in all modules, with new fluxcon tables for each module. As a result of the change in bias made for the LH module at the beginning of IRS Campaign 25, we now provide a different fluxcon table for data taken from Campaign 25 onwards.
As a result of improved flux calibration, we discovered a phenomenon we have called the "teardrop" effect in the SL module. The SL fluxcon tables differ from previous pipeline processing, in that the teardrop enhancement is not fitted out by the fluxcon tables. The resulting fluxcon table is much more linear over the SL1 module. Users are warned that as a result of the new fitting procedure, the teardrop effect will be more obvious than it was in past processing (previous processing partly masked the effect of the teardrop to the detriment of our flux calibration). The cause of the teardrop effect is not fully understood yet by our team.
Linearity Coefficients post LH bias-change
We have modified the linearity coefficients use to correct non-linearities in the ramps for data taken in LH after the bias change (Campaign 25 and onwards).
Browse-quality Automatic Background Subtraction for Low-res modules in Stare-mode only
We have implemented an automated background subtraction process for all low-res data taken in staring mode. The process operates only on co-added data, and subtracts adjacent nod positions. Spectra are extracted automatically from these new products. (See IRS DATA Handbook for details). We caution users that this process is not necessarily the optimal way to do background subtraction in a number of cases. Firstly if the source is extended over a significant part of the slit then obviously the automated nod subtraction will not work. Secondly, for very bright point sources, the extended PSF from one nod position could extended far enough at the long wavelength end of the spectrum to compromise the background subtraction. The greatest benefit of the automated subtraction will be for faint point-sources with no other confusing source in the slit. Useful filenames have the following suffixes: bksub.tbl (irs-tuned final nod-subtracted spectrum), bksub_unc.fits (uncertainty file). If you have downloaded your S13 data prior to 02 Feb 2006, it may have been missing the post-BCD background subtracted staring mode data. If this is the case, and these data are needed, you should download the files again.
Coordinates and wavsamp_offset files
S13 SPICE has the capability to view the coordinates of sources which are within the slit. For this purpose, it utilizes the b?_wavsamp_offset.fits and b?_wavsamp_omask.fits cal files. The cal files, specifically the b?_wavsamp_offset.fits files, which you get when downloading your pipeline processed data from the archive have the incorrect sign convention which will result in the coordinates being incorrect. Therefore, S13 SPICE should only be used with the cal files which come with the SPICE release rather than the archived cal files.
Row-Droop Correction Applied to SL module
As discussed in detail in the notes for the S12.0 pipeline release (below) we were not correcting fully for the effects of row-droop. This problem has been fixed in S13. The correction is only applied to SL because of the high charge loading on the SL peak-up arrays. In the other modules, no correction is necessary.
Browse-quality IRS Peak-up Imaging through the MOPEX image mapper
Unlike S12, we now pass data that was obtained under IRS Peak-up Imaging mode through the MOPEX mosaicking software and generate browse-quality peak-up images. The files are named with the following suffixes: b_mos.fits and r_mos.fits (blue and red peak-up mosaics), b_unc.fits and r_unc.fits (uncertainty files) and b_cov.fits and rr_cov.fits (coverage maps).
9.5.1.10 S12.0 (Release Date: Jun 2005)
All IRS data taken in Nominal Ops campaigns 1-20, and SV campaigns N-R have been re-processed with S12 and are in the process of being archived. Check on this page to see when your favorite campaigns have been archived and are available for download.
Memo on IRS uncertainties (October 2005)
There are three significant changes to the data processing for S12. These are:
1. Wavelengths: The wavelength solutions (wavsamp files) for all modules have been changed. This was primarily to correct some inter-order offsets that were apparent in the S11 data. These changes include wavelength shifts (e.g. for SL1) and changes to the dispersion (e.g., for SH, LH LL and SL2). Lines will shift (from their S11 positions) by 0.01-0.14 microns, depending upon wavelength and module.
2. Flux calibration: Small changes to the Short-High and Long-High flux calibration (fluxcon files) were made in accordance with the wavelength changes described above. These should reduce order to order variations seen in some S11 data.
3. Short-Low stray light: The correction for stray light in the Short-Low module (see the Observers Manual and Pipeline Handbook) has been modified to include only an exponential fall-off from the Peak-up arrays for S12. Previously, this stray-light correction also included a small "pedestal" term in addition to the exponential. The removal of this pedestal occasionally caused unwanted horizontal striping in Short-Low data. These stripes should not be present in your S12 data. Observers should be aware that although the net effect of this modification is positive (data processed through the S12 are of generally superior quality to S11), the fact that we no longer remove a "pedestal" means that a minor effect called "row droop" will be present in the final spectra. As we discuss below, this will have almost no impact on users if they perform a background subtraction of SL data (this is highly recommended in most observing strategies). However, in the unlikely event that background subtraction is not performed, observers should be aware of the possible consequences of uncorrected row-droop.
"Row-droop" is an additive (pedestal-like) signal which can affect all the ramps along a given row of an image by a constant factor which is proportional to the total charge seen on that row of the detector. The Short-low module is the most likely detector affected by the problem because the peak-up arrays can contain a significant signal--and this signal can cause an increase in the flux seen along that entire row--especially when observing medium and high background scenes. Because we are no longer removing a pedestal from SL data in the stray-light correction, row-droop will be present in your S12 Short-Low BCD data at some level of significance. It will manifest itself as an apparent increase in the flux in SL2 and SL1 in regions where the peak-up arrays affect those rows. In contrast, the pedestal will NOT BE present in rows between the two peak-up arrays--an effect that could cause the observer to think they had faint absorption there. The spectral region that might show an apparent flux deficit is between 5.5-5.9 microns and 11.2-11.8 microns--the wavelength regions corresponding to the rows in SL2 and SL1 that are between the peakup arrays. Sky subtraction will remove this effect. However, observers of extended sources should take care in interpreting low-level spectral variations at these wavelengths in their Short-Low S12 data if no appropriate sky subtraction is performed. We will characterize and remove this pedestal in future pipeline releases. Observers are encouraged to compare their SL data before and after sky subtraction to gauge the relevance of this effect on their science.
9.5.1.11 S11.4.0 (Release Date: Feb 2005)
Most of the changes were uplink-related (e.g., Spot-related) and thus do not affect the data. However, the LH dark calibration files (which are updated every campaign) have changed and so depending on whether or not you have data from the IRS LH module, you might have updated dark files.
9.5.1.12 S11.0.2 (Release Date: Dec 2004)
Complete reprocessing of IRS campaigns using S11.0.2 software. Modifications include:
1. Completely revised Droop Coefficients for all modules.
2. Completely revised linearity corrections for all modules.
5. All LH data are now processed with campaign derived darks (to partially mitigate against time-variable warm pixels).
6. Fix to stray-light correction algorithm and operation of module that was causing problems "banding" across spectral orders.
7. Fix to "bright source drop-out" problem that cause trouble with 4-sample ramps of very bright sources. This problem caused sources which were only measured with two independent samples to have spectra whose fluxes were lower than they should have been. this problem was fixed in this release.
8. Fix to "*_spec2.tbl files not being archived" problem.
9.5.1.13 S11.0 (Release Date: Nov 2004)
Complete reprocessing of IRS campaigns using S11 software. Modifications include:
1. Addition of new Peak-up mapping pipeline to support GO2 Peak-up mapping capability.
2. Fix to "high-flux drop out" problem seen in 4-sample spectral observations in which the 3rd and 4th samples are saturated because the target is so bright. RADHIT and IMAGEST modules can now handle a single-slope measurement required to capture information from the unsaturated samples. This problem ONLY affects 6-s RAW data on extremely bright sources.
3. All S11 data incorporates newly measured "Droop" coefficients and refined and improved non-linearity corrections for all 4 modules.
4. As a result of 3, new flat field/spectral response corrections and flux conversion tables are also applied.
9.5.1.14 S10.5 (Release Date: Sep 2004)
A complete reprocessing of IRS campaigns using the existing S10 pipeline but with the following change resulting from new calibration products:
1. Improved performance of pipeline slope estimates in regions close to A/D saturation.
9.5.1.15 S10.5 (Release Date: 8 Aug 2004)
A complete reprocessing of IRS campaigns incorporating the following changes:
1. Fix to ramp timing problem for slope estimates for spectral data.
2. New higher-quality "flat-fields" (i.e. spatial flats/2-d RSRFs)-- improvement mainly in spatial quality along the slit.
3. New associated fluxcon tables.
9.5.1.16 S10.0 (Release Date: 14 Apr 2004)
Complete processing of IRS campaigns using best available calibration products available at that time.
1. Nod to nod variations in flat fields no better than 7-10%.
2. Problem with spectral data slope estimatation especially for very short ramps. This problem was fixed in S10.5.
9.5.1.17 S9.5 (Release Date: 20 Feb 2004)
1. Incrementally improved calibration products after analysis of post-launch solar-flare activity and in-flight performance.
