2MASS External Review Board - February 2003
I. Data Processing Overview
Data processing for 2MASS was conducted using the highly automated 2MASS Production Processing System (2MAPPS) which converted each night's raw imaging data from each telescope into calibrated images and extracted source lists. Processing was done in two stages. Preliminary processing was carried out while the survey was ongoing using a version of 2MAPPS that was allowed to evolve and improve continuously as our knowledge of the telescopes, cameras, detectors, atmosphere and the near infrared sky improved. The 2MASS Sampler, First and Second Incremental Data Releases were derived from the results of the preliminary data processing. A second, complete processing of all survey data following the completion of the survey observations was designed to reduce all survey data with a single, stable software platform, 2MAPPS v3.0. 2MAPPS v3.0 incorporated lessons learned during the preliminary processing and feedback provided by users of the Incremental Data Release products. It also made use of reference datasets not available earlier, such as the Tycho 2 astrometric reference catalog, and information gathered from analysis of the complete survey data set. 2MAPPS v3.0 also contained numerous upgrades that improved the general quality and yield of data from the survey. Finally, 2MAPPS v3.0 included more extensive Quality Assurance tied explicitly to metrics derived from scientific investigations of the data.
II. Upgrades in 2MAPPS v3.0
2MAPPS v3.0 included all of the modifications to the major subsystems
proposed during the
June 2001 External Review Board Meeting, except for one - implementation
of active deblending in the profile-fit photometry routines.
Numerous other modifications were made to fix minor bugs, improve
runtime efficiency, increase data yield, and improve quality assessment
of the end products. The list below provides a summary of all top level
improvements that were integrated into 2MAPPS v3.0.
- Instrumental Calibration
- Use of 2-dimensional saturation map for each array. A constant saturation threshold value for all pixels in each array was used during preliminary processing.
- Best time-Dependent Canonical Dark Images and Flat-field Images generated using all survey data. Flat-field images incorporate corrections for cross-scan biases observed in point source photometry.
- Atlas Images
- Meteor trail detection and blanking for Atlas Images implemented.
- Improved frame background "tilting" in Atlas Image construction to deal with high-order spatial structure in frame backgrounds.
- Atlas Image backgrounds provide better represenation of true Poisson noise for setting detection thresholds. Result is that Atlas Image backgrounds may not match smoothly in-scan.
- Single pixel coverage allowed into Atlas Images to minimize "holes"
- Point Source Characterization
- Improved noise estimator for establishing source detection thresholds. Less sensitive to low frequency background structure.
- Saturated R1 star detection, centroiding and brightness estimation implemented using 1-dimensional radial profile fitting.
- Improved set of point-spread-functions and variance maps developed for all arrays and all hardware periods. Results are better uncertainty estimation vs. brightness and chi-squared statistical distributions.
- Used seeing-dependent look-up table values for aperture photometry curve-of-growth and normalization of R1/R2 photometry. This minimized errors due to small number statistics in empirical derivation on scan-by-scan basis.
- Extended Source Characterization
- Higher-order background fits used for extended sources measurements to mitigate H-band airglow structure
- Training set for decision tree used for extended source classification expanded using data from full survey.
- Improved robustness of photometry in crowded regions by using an iterative fit/subtraction of foreground stars.
- Galaxy completeness improved in high source density regions by ~0.5 mag through better multiple star discrimination.
- Specialized processing for Large Galaxies.
- Extended source coverage maps generated.
- Astrometry
- Higher density Tycho 2 Catalog used a primary astrometric reference catalog.
- Measure and momitor focal plan distortion and included in source astrometric reconstruction.
- Used position/uncertainty correction tables derived from preliminary processing to propagate astrometric solutions across scan boundaries. This improves performance in regions with low astrometric reference and 2MASS star densities.
- Artifact Identification
- Improved identification of artifacts from bright star artifact enabled by saturated star brightness estimation and updated geometric algorithm parameter derivation.
- Photometric Calibration
- Derived improved secondary standard list compiled from full survey. Best possible internal consistency and rejection of variables.
- Time variable atmospheric extinction model used (varies by month) derived from global calibration solutions from entire survey.
- Employed higher order fits to nightly zero point offset measurements.
Piece-wise fit used in the J-band, linear fits used for
the H and Ks bands). These fits produce lowest overall
residuals for calibration stars.
- Minor Planet Identification
- Newest ephemerides used for asteroid and comet identification.
- Planetary satellites identified.
- Quality Assurance
- Greatly expanded automated quality review of nightly processing. Introduction of QA diagnostics based on scientific metrics.
III. Active Deblending
Active deblending refers to the process of fitting two or more point-response-function components to a single detection if the chi-squared goodness-of-fit for a single component is above some threshold. Although the testing and analysis of the active deblending software showed that it worked well under most conditions, the Science Team opted to not implement it in final processing. We were led to this decision primarily because the slight astigmatism in the 2MASS optical system produced point-spread-functions that varied slightly across the focal plane. The degree of variation was exaggerated when the telescope was out of focus, and this could have led to preferentially deblending single objects at the edges of the scans. Small focus errors were not uncommon early during nights on nights with large diurnal temperature differences and rapid cooling after sunset when the temperature drop was faster than the response of the telescope's autofocus routines.
IV. Final Processing Summary
Final 2MASS Survey processing using 2MAPPS v3.0 began on 2 August 2001 and was completed on 6 February 2002. All survey data acquired under photometric conditions were included in the final processing. This included scans of 70,712 Tiles taken on 1415 survey nights (691 northern and 750 southern) spanning 7 June 1997 UT to 15 February 2001 UT.
Final processing generated Working Source Databases containing data for 1,314,981,867 Point and 2,590,500 Extended source extractions, and an Atlas of 4,879,128 calibrated FITS images, from which the All-Sky Release Products were drawn.
Last Updated: 31 January 2003
R. Cutri - IPAC