2MAPPS v3.0

2MAPPS v3.0 Design, Implementation and Testing

I. Overview

The stringent quality requirements for 2MASS data products and the requirement to release provisional 2MASS data to the astronomical community as early as possible, led to the two-phase data processing strategy. Preliminary pipeline data processing was started with the 2MASS Production Processing System (2MAPPS) version 2.0 shortly after survey observations started. 2MAPPS v2.0 was demonstrated to produce products that met the survey's Level 1 Science Requirements under most conditions, with the possible exception of global uniformity. With some care, preliminary incremental release catalogs and Atlas images have been produced from 2MAPPS v2.0 processed data and distributed to the community.

2MAPPS v2.0 has been evolving and improving steadily during the life of the survey as our understanding of the telescopes, cameras, detectors, atmosphere, and near infrared sky has improved. The final uniform processing of the 2MASS data following the end of survey observations is designed to reduce all data with a stable software platform, 2MAPPS v3.0, that incorporates the knowledge accumulated during the preliminary processing to produce highly uniform final products. Processing all data with a single pipeline version couples with a single data acquisition mode and identical telescopes and camera to add to the uniformity of the data products.

At minimum, 2MAPPS v3.0 can be equivalent to the state of 2MAPPS v2.0 at the completion of the preliminary processing, with the correction of any remaining bugs or deficiencies. This would achieve the goal of software uniformity for the final processing. However, the final processing affords the opportunity to include improvements to the pipeline that further improve the quality and yield of the survey data. This page discusses the overall constraints on improvements for 2MAPPS v3.0 imposed by hardware, runtime and schedule resources, the prioritization and cost/benefit analysis of possible upgrades, and the schedule for 2MAPPS v3.0 development.

II. Boundary Conditions on Possible Improvements

1. Runtime

Final processing of 2MASS data must be completed in approximately one calendar year or less to allow sufficient time for final product generation and validation. Therefore, the product of reprocessed data volume and pipeline runtime must be less than one year.

With the current budget, there are no resources available to upgrade the production hardware significantly. We expect to have the equivalent of 125% of the current CPU power because the two E250 machines currently being used for tape validation and 24hr QA duties will be allocated to production for the final processing. The default final production cluster will be the equivalent of five E450 servers. Additional hardware augmentations are possible at the expense of analysis and quality assurance workforce if it is deemed necessary. The throughput of 2MAPPS v2.x is one average night (approximately 65 science scans) per 24 hr period on one E450 machine. Thus, five average survey telescope-nights can be processed per 24 hr period at the current 2MAPPS runtime.

The minimum data volume for final processing is equivalent to one scan per Tile on the sky, or 59,650 scans. This is the approximate equivalent of 918 average survey nights. Partial nights are processed with slightly lower efficiency because of the overheads associated with the nightly processing unit such as calibration and quality assurance. There are large numbers of multiply-observed tiles, special filler tiles, test tiles, etc. and there will undoubtedly be demand to process these for science and validation reasons. An estimate of the maximum data volume for final uniform processing is 76,924 scans, the total number of scans with quality >0, as represented in the SIT file date 001109. This is the equivalent of 1183 average telescope-nights

At the 2MAPPS 2.x processing rate, these two limits indicate 184-237 days for the final processing, at peak operational efficiency. Keeping the worst-case estimate at <365 days means that 2MAPPS v3.0 cannot exceed 1.5 times the 2MAPPS 2.x runtime without hardware augmentation. Allowing for 20% schedule margin, a bare minimum considering that we have not taken into partial night overheads or processing interruptions of any kind, the maximum allowable runtime increase should be 130% of 2MAPPS v2.x without hardware upgrades.

2. Cost-Benefit Analysis

Most of the modifications to the existing pipeline for the migration to v3.0 will fit into one of these categories:

Fixes to known bugs or deficiencies that compromise Level 1 Requirements
Addition of diagnostics to improve quality validation and/or final product generation with existing algorithms
Incorporation of a priori knowledge of the 2mass infrared sky into the final processing to yield benefits in the processing, uniformity and data quality
Modifications to improve efficiency and runtime of current algorithms
Improvements to increase yield of data that meet Level 1 Requirements
Modifications to improve data quality beyond Level 1 Requirements

These categories are listed in approximate priority order relative to achieving the 2MASS mission objectives. All suggested fixes and modifications will need to be accompanied by an estimate of the cost (and time) necessary for the changes, and appropriate justification of why this modification is necessary. This will be especially important for category e changes.

Ultimately, which modifications are implemented in 2MAPPS v3.0 should be determined by the results of the cost-benefit analysis.

3. Supporting Analysis

Most of the recommended modifications for 2MAPPS v3.0 will require considerable supporting analysis to test and optimize new algorithms, and new code. The cost factor of each proposed upgrade should take these analyses into account.