IV. 2MASS Data Processing

2. Instrumental Frame Calibration

a. Frame Readouts

The 2MASS arrays are read out non-destructively twice during each exposure, after 51 ms and 1.3 s, in stages dubbed "Read_1" and "Read_2", respectively (see III.1.b.ii). The primary pipeline operations are performed on the "Read_2 - Read_1" frames for which constant electronics systematics cancel out. Photometry for bright stars which saturate in Read_2 is derived directly from the Read_1 measurements.

Data frames in all bands are corrected for electronics bias, array responsivity (flatfield), and short term frame biases. Pixels in the raw frames whose intensities exceed linearity thresholds are marked as saturated. Each of these characterizations and the overall calibration sequence are detailed in the following sections.

b. Darks/Electronics Bias Characterization

Instrumental characterization data is acquired during nearly every night of 2MASS operations. These data include series of dark measurements (frames acquired with a cold shutter obscuring the detectors).

Nightly electronics bias correction images in each band are generated in the pipeline processing by combining all of the dark sequence frames. Both Read_1 and Read_2 measurements are made, with Read_2-Read_1 darks applying to the production frames. The nightly darks are seen to be stable spanning periods of weeks to months with characteristic structures in each band that generally scale directly with overall darks level in the frame.

To maximize consistency for final production, the nightly darks are grouped into blocks of time with consistent behavior and averaged into "grand canonicals" that are applied to all data during that period. Breaks between "grand canonical" darks typically correlate with hardware activities at the telescopes (including summer shutdowns in the north) that result in slight changes to the electronics characteristics. Break periods are identified primarily by discontinuous jumps in median dark level in one or more bands. Figures 1 and 2 show the running dark medians for the North and South respectively, with corresponding breaks marked by vertical lines. Northern summer shutdowns are indicated by grey bars.

Figure 1 - Median J, H and Ks dark exposure pixel values plotted as function of time for the northern 2MASS camera. Vertical lines denote boundaries of the periods for which "canonical" dark frames were generated. Figure 2 - Median J, H and Ks dark exposure pixel values plotted as function of time for the southern 2MASS camera. Vertical lines denote boundaries of the periods for which "canonical" dark frames were generated.

A total of 16 darks periods are utilized in the North, with only 4 warranted in the South. Summary tables follow showing date ranges for the periods. Columns are the period label, the date range (using a YYMMDD format), which bands showed systematic changes from the previous period, and a brief note indicating events associated with the end of the period. 

NORTHERN HEMISPHERE DARKS

  Dates         Bnd  Notes
--------------------------------------------------------------------
a 970000 970830 ---  summer shutdown
b 970901 971114 JHK  work during rain at telescope for several days
c 971115 971224 J    holiday shutdown
d 971225 980411 JH   zener diodes added (read1 changes)
e 980412 980719 JHK  summer shutdown
f 980720 980924 JHK  Leach test at start of night
g 980925 981014  H   camera pumped down (only 3 days of darks; no break)
h 981015 981022  H   Leach replaces GATIR electronics
i 981023 981118 J K  Leach eprom change (actually before obs on 18th??)
j 981119 990427 JH   Leach software upgrade
k 990428 990612 JH   slight warmup of camera
l 990613 990720 J K  summer shutdown
m 990721 991202 JHK  H banding problems show in darks?
n 991203 000322  H   camera pumped down
o 000323 000724   K  summer shutdown
p 000725 020101 JHK  end of ops



SOUTHERN HEMISPHERE DARKS

  Dates         Bnd  Notes
--------------------------------------------------------------------
a 970000  990117  ---  electronics maintenance
b 990116  990226  JHK  Leach installed
c 990227  990506  JHK  Leach eprom upgrade
d 990507  020000  JHK  end of survey

c. Flatfield/Responsivity Characterization

Responsivity images (multiplicative gain corrections) are derived from the measurements of the rapidly dimming or brightening twilight sky by charting the relative change in intensity seen in every pixel in response to the changing illumination level. The resulting pixel-by-pixel responsivity images are normalized to have a median of unity.

Initial operational production utilized a running 5-night average of "good" flats (ones obtained under proper illumination levels uncontaminated by clouds), producing nightly "canonical" flats.

Analysis of the full flats library for both hemispheres indicates that many of the flatfield drifts seen across timescales of weeks to months are seasonal in nature and more likely reflect changing illumination patterns within the observatories rather than actual drift in electronics responsivities. Therefore more uniform canonical flats averaged over longer periods have been adopted give more consistent cross-scan biases (illumination-derived variations in flats across the detector that do not correlate to actual responsivities can contribute to photometric biases as a function of horizontal position within the image).

Similar to the handling of the darks, final processing has employed "grand canonical" flats derived from the medians of all good flats within the corresponding periods. Flats "grand canonical" periods were evaluated on a band-by-band basis (i.e. a change in responsivity seen only in H band on a given date would trigger a new H period but not break the J & Ks periods. Breaks are triggered only by significant changes in responsivity pattern within a given detector, often associated with hardware work at the observatories.

Each band of the north warrants 6 periods (though they are not exactly the same for any two bands; overall there are 8 separate periods) while all south bands fall into the same 4 periods. Flats periods are summarized in the table below. Columns are the period label, the date range (using a YYMMDD format), and the list of periods by band. 

NORTHERN HEMISPHERE FLATS

   Dates          Bands
----------------------------
a  970000 970802  J1  H1  K1
b  970903 970922  J2  "   " 
c  970927 971026  J3  "   K2
d  971027 980918  J4  H2  K3
e  980919 981020  "   H3  K4
f  981021 990720  J5  H4  K5
g  990721 000901  J6  H5  K6
h  000902 050000  "   H6  " 


SOUTHERN HEMISPHERE FLATS

   Dates          Bands
----------------------------
a  970000 990117  J1  H1  K1
b  990118 990226  J2  H2  K2
c  990227 990624  J3  H3  K3
d  990625 050000  J4  H4  K4

d. Frame Calibration Procedure

The basic frame calibration procedure follows typical astronomical conventions. The process is summarized in the following list and in Figure 3.

The final step compensates for any short term effects not characterized by the long-term darks/flats corrections. Within each scan, a series of additive sky illumination corrections are derived by creating -trimmed averages for blocks of at least 42 dark-subtracted, flat-fielded sky frames. The trimmed averaging rejects any sources within the frames and yields a measurement of residual dark-sky illumination patterns on the detectors within each block. This so-called "sky offset" frame is then subtracted from each input frame, resulting in a data frame ready for source detection and combination into the final survey Atlas Images. The background levels of the final instrumentally calibrated frames correspond to the original sky levels.

Figure 3 - Instrumental frame calibration procedure shown for a sample Ks-band image.

e. Non-Linearity/Saturation Flagging

No linearity corrections were applied to 2MASS frame data. Instead, pixels in the raw frame data that exceeded a threshold intensity value defined to be the point where the pixel response function deviated from linearity by >1% were flagged as "saturated."

Saturated pixels were flagged by setting the least significant bit of each pixel's mantissa to a zero or 1 to indicate an unsaturated or saturated state, respectively. The lowest order bit was not altered by subsequent frame calibration steps. This status bit was used by the astrometric and photometric measurements segments of the pipeline to detect saturation.

i. Saturation Threshold Method

Saturation thresholds were derived for each pixel in each detector array using pixel response curves measured from dawn and dusk twilight sky exposures from several nights. A typical response curve for the southern observatory J-band array is shown in Figure 4. The small dots represent the median Read_2 pixel intensities which are plotted as a function of the median dark-subtracted Read_1 intensities in each frame in the twilight exposure sequence. Note the hard saturation around 56000 DN (digital number) and the slight negative curvature in the middle. The smooth dashed and dotted lines are overlaid linear and quadratic fits, respectively, to the frame median data in the 15000-47000 DN range. For each pixel i, a linear fit to the low-light part of the pixel response curve (yi), as a function of light level (x), was compared with a quadratic fit to the response curve in the vicinity of the threshold. The saturation (or non-linearity) threshold is estimated for each pixel as the point where the low-light linear fit and the high-light quadratic differ by 1%.

Saturation data for each array were obtained by combining several single-night estimates, first (a) subtracting the median dark for the night, then (b) combining the available thresholds for each pixel to obtain a single best estimate of the "canonical" (i.e., dark-subtracted) threshold. Finally, to obtain the best threshold estimate for each night, the appropriate nightly dark was added back into the canonical threshold. Pixels for which no good estimates are available are set to a default value, which is the 1%-tile value for all other good pixels in the same array quad, and with the same column parity (i.e., odd or even).

Figure 4 - Southern camera J-band pixel response function.

ii. Saturation Threshold Statistics

Table 1 shows for each detector array, (a) the number of nights N combined in the threshold generation process (note that N', the number of available nights for a given pixel, may be less than N, since a pixel may fail in some nights and not in others); (b) the average of the difference between the maximum and minimum threshold (over N') for each pixel, (c) the average estimated standard deviation of the threshold for each pixel, and (d) the fraction of good pixels in the map. The estimated pixel standard deviations are derived from the scatter of the available good thresholds for each pixel for N'. The fraction of good pixels in a single night's analysis is typically 90% to 98%. Combination of nights, even when only two nights are available, gives >97% good pixels for all six of the original detectors:

Table 1 - Pixel Saturation Map Statistical Properties
North Camera: # Nights: Pixel Diffs: Pix.Sd.Devs: FrcGoodPix:
J 2 1661.56 1174.90 0.997
H (OLD) 2 2319.64 1640.23 0.971
H (NEW) 1(4) 229.26 149.40 0.944
Ks 2 710.96 502.72 0.983
South Camera: # Nights: Pixel Diffs: Pix.Sd.Devs: FrcGoodPix:
J 4 1843.94 843.94 0.998
H 3 2483.36 1333.29 0.993
Ks 2 567.10 401.00 0.978

The new northern H-band array is exceptional in several respects. Because its pixel properties vary strongly across the array, it was not possible to find a set of fit intervals and accept parameters that gave more than about 60% acceptable pixels, the rest being set to the default. Instead, four fits have been made using data from one night (9/29/1999 UT), varying the parameters to optimize the analysis for different subsets of pixels. This is the meaning of the "1(4)" entry in the table. The four separate threshold maps were then combined, taking the good pixels from each. Because data from only one night was used, the Pixel Diffs and Pix.Sd.Devs are unusually low; this agreement provides some reassurance that the different analyses have not produced wildly varying incorrect results. Note however that many pixels have in fact only a single analysis run for which they passed the quality control tests. (This, incidentally, accounts for the median in the Pix.Sd.Devs plot being zero: over half the pixels had only one good sample.)

Examination of summary plots of the data versus the low-light linear fit show that the yi differ from the linear fit by 2% at approximately 5000 DN greater than the point at which the difference is 1%. Thus the photometric error due to these thresholds is typically 1% for stars just below the threshold, and should be <2% for essentially all pixels, considering the uncertainty in threshold estimation.

iii. Saturation Threshold Maps

The figures below show, for each array at each observatory, (a) a histogram of the canonical saturation thresholds (i.e., before addition of the appropriate DARK for the night); (b) an image (stretch: 30K black; 45K white; blue-green-yellow-red in spectral order in between, with midpoint 37.5K green) of the canonical saturation thresholds, and (c) a histogram of the estimated pixel standard deviations.

NB: Incorporation of the nightly dark both (a) shifts, and (b) broadens the threshold histograms. In the histograms, the vertical dotted line indicates the median; the vertical dashed lines the 10%-tile and 90%-tile limits; and the vertical dash-dotted lines the 1%-time and 99%-tile limits.

North Observatory
Band Threshold Histogram Threshold Image Pixel St.Dev
J
H (Original)
H (New)
Ks

South Observatory
Band Threshold Histogram Threshold Image Pixel St.Dev
J
H
Ks

[Last Updated: 2005 December 5; by R. Hurt, E. Kopan, W. Wheaton and R. Cutri]

Previous page. Next page.
Return to Explanatory Supplement TOC Page.