Appendix 5. Working Database Source Reliability Estimation

2. Point Source Reliability Estimation

Each extraction in the Survey Point Source Reject Table and 6x and Calibration Point Source Working Databases (PSWDBs) is assigned a reliability score, rel, that is related to the probability that the extraction is a detection of a real astrophysical source at the time of the 2MASS observation. The reliability score is a single character with a value A-F that maps into a probability range according to Table 1.

The algorithm for assigning the reliability score to point source extractions is developed using a large "truth table" of WDB extractions with independently determined reliability to examine how fractional reliability of the ensemble changes with source parameters such as brightness, signal-to-noise ratio (SNR) and band-detection combinations. The empirical relationships between reliability and these parameters are then converted into a matrix of primary scoring rules that are used in conjunction with artifact flagging, frame detection statistics and profile-fit photometry chi-squared thresholds to assign reliability scores to each point source WDB extraction.

Table 1 - Estimated Range of Probabilities for Each Value of the Point Source Reliability Score, *rel*
rel	Probability Range
A	P>90%
B	80<P<90%
C	70<P<80%
D	50<P<70%
E	20<P<50%
F	P<20%

a. Point Source "Truth Table"

The "truth table" used for the point source reliability parameterization was generated from the set of extractions in the Survey Point Source Working Database (PSWDB) that fall within the areas covered by the 2MASS calibration tiles. The reliability of each extraction in the truth table is determined using detection statistics from the hundreds to thousands of independent scans of each calibration tile (i.e. how many times a source was detected out of how many times it was observed). The truth table was constructed as follows:

The rectangular region within each of the 35 primary 2MASS survey calibration field that was covered by all available scans was first tabulated (i.e. see the calibration scan footprint maps in Table 1 in A.4.2.b).

All extractions in the Survey PSWDB that fall within the full-coverage regions of each calibration fields, that are not identified as an artifact detection (cc_flg NOT MATCHES '*[A-Z]*') and that have a valid photometric measurement in at least one band (rd_flg MATCHES '*[123]*') were selected as truth table members.
The Calibration Scan PSWDB was then searched to find all extractions within 1.5'' of the position each truth table source.

For each truth table source, the band-by-band detection fraction (number of times detected divided by the number of times scanned) was computed, along with the mean and RMS (root variance) position and fluxes using the matching apparitions in the Calibration PSWDB. For example, Figure 1 shows the J, H and K_s detection fraction for all truth table sources in the 90301 calibration tile plotted as a function of mean source brightness. Bright sources are detected in virtually every available scan. The detection rate/completeness drops off quickly with decreasing brightness for sources fainter than J~16.5, H~15.5 and K_s~15 mag.

Each truth table extraction was classified as a real or false detection based on the number of times it was detected in the calibration scans. Figures 2 and 3 show histograms of the detection counts in each band for truth table sources in the 90301 field. The strong peak at near 3430 detections corresponds to reliable, high SNR sources that are detected in nearly every scan. The peak near ~10 detections is produced by chance positional correlations between unreliable, randomly distributed noise extractions. These corresponds to the "pile-up" of faint truth table candidates with very low detection fraction seen in Figure 1. Visual examination of the images of truth table sources detected at least 50 times in this field confirms that all are real object. Therefore, all sources with >50 detections are classified as reliable and those detected <50 times are unreliable. The truth table sources in all of the remaining calibration fields were classified using similar, well-defined low detection rate peaks.

Figure 1 - Detection fraction versus magnitude by band for sources in the 90301 calibration field that was scanned a total of 3430 times. Figure 2 - Detection frequency histograms for sources in the 90301 calibration field. J distributions are shown in blue, H in green and K_s in red. Figure 3 - Close ups of the low and high detection frequency peaks in the distributions shown in Figure 2.

b. Parameterizing Reliability

The net reliability for an ensemble of truth table extractions is given by the ratio of the number of reliable extractions (N_rel) to the total number of extractions (N_tot=N_rel+N_unrel):

R = N_rel/(N_rel+N_unrel)

Figures 4 and 5 shows the histograms of the number of reliable and unreliable truth table sources as a function of magnitude and photometric measurement uncertainty bins from four high latitude calibration fields (90301 - 3430 scans, 90266 - 2776 scans, 90272 - 1973 scans, and 90330 - 1192 scans), where source reliability has been classified as described above. Real sources are shown in black and spurious extractions in red.

The magnitude at which a particular sensitivity is achieved in the 2MASS observations can vary by nearly one magnitude because of variations in atmospheric seeing, background and throughput conditions (see VI.2). Consequently, photometric measurement uncertainty (or equivalently SNR) is a better parameter by which to characterize reliability than magnitude. Figure 6 shows the differential net reliability distribution as a function of photometric uncertainty derived from the distributions in Figure 5.

Figure 4 - Log histograms of the number of reliable (black curves) and unreliable extractions (red curves) as a function of magnitude for the Survey PSWDB truth table extractions located in the area covered by the 90266, 90272, 90301 and 90330 calibration fields. Figure 5 - Log histograms of the number of reliable (black curves) and unreliable extractions (red curves) as a function of photometric measurement uncertainty for the Survey PSWDB truth table extractions in the 90266, 90272, 90301 and 90330 calibration field areas. Figure 6 - Net differential reliability as a function of photometric measurement uncertainty for the Survey PSWDB "truth table" extractions in the 90266, 90272, 90301 and 90330 calibration field areas.

Figures 7-11 show the dependence of reliability on photometric uncertainty derived using the Survey PSWDB truth table extractions in all of the 2MASS calibration fields for different band-detection combinations. Figure 7 shows the reliability for 3-band detections, Figures 8-10 show 2-band detections (JH, JK_s, H_s, respectively), and Figure 11 shows reliability for single-band detections. In each figure, the top panel shows the net reliability curves, with J-band shown in blue, H-band in green and K_s-band in red. The bottom three panels in each figure show log histograms of real (black) and spurious (magenta) source counts in the three bands as function of photometric uncertainty.

The 3-band detection reliability curve shown in Figure 7 indicates that extractions with valid J, H and K_s are virtually always real, independent of source SNR. Low SNR extractions in one band are frequently accompanied by a higher SNR extraction in another band that "pull-up" their net reliability. Two-band detections are also highly reliable up to an uncertainty of msig~0.2 mag (SNR~5), for the same reason. However, 2-band-detected extraction reliability decreases systematically towards lower SNR values. Single-band detections have the poorest overall reliability in the Survey PSWDB, and their net reliability also declines rapidly with decreasing SNR. J-band-only detections are the most common real single-band sources because the survey's instrumental response was highest in the J-band and the atmospheric backgrounds are lowest. This is augmented by the fact that the majority of sources detected by 2MASS are Milky Way stars, most of whose spectra decline between 1.2 and 2.2 microns.

Figure 7 - 3-band extractions Figure 8 - 2-band (JH) extractions Figure 9 - 2-band (HK_s) extractions Figure 10 - 2-band (JK_s) extractions Figure 11 - Single-band extractions

Net reliability vs. photometric measurement uncertainty ([jhk]_cmsig) for survey PSWDB truth table extractions in all calibration tiles. Top panels shows J (blue), H (green) and K_s (red) reliability curves. Bottom three panels show xmsig histograms of reliable (black) and unreliable (magenta) extractions in J, H and K_s.

c. Other Reliability Indicators

Other attributes assigned to point source extractions during pipeline data reduction are effective means to identify unreliable detections, and are incorporated into the WDB reliability scoring. These include artifact flagging, frame detection statistics, and for non-saturated sources, the chi-squared goodness-of-fit parameter from point source profile-fitting photometry, and all were used to filter out unreliable sources during construction of the All-Sky PSC and 6x-PSC, as described in V.3 and A3.6.c. See also the description of the attributes of unreliable sources found in the 2MASS All-Sky PSC in I.6.b.vi.

d. Point Source Reliability Scoring Algorithm

The reliability score for point source WDB extractions was determined initially using the number of bands-detected, the photometric uncertainty (and SNR), and artifact flagging. The initial score was then revised if the extraction possesses characteristics of known spurious detections (e.g. residual cosmic rays, meteor trails, hot pixels, edge flares, etc). Note that because reliability is determined as a function of photometric measurement uncertainty/SNR rather than measured flux, the same relationships can be used for the Survey/Calibration PSWD and the 6x PSWDB extractions.

For the purposes of reliability scoring, an extraction is considered to be detected in a band only if rd_flg MATCHES '[123]' in that band. If an extraction has no valid detections in any band, then it is assigned a reliability score of rel='F'.

i. Initial Score

3-Band Detections

rel='A' unless extraction is flagged as an artifact (cc_flg MATCHES '[A-Z]') in all three bands

rel='F' if extraction is flagged as an artifact in all three bands

1- and 2-Band Detections

rel='F' if extraction is flagged as artifact in any detected band (cc_flg MATCHES '*[A-Z]*'), or

rel='F' if extraction is single-band rd_flg="1" or "3" source

Otherwise, reliability is set using the SNR in each detected band, where the SNR in each band is taken to be the larger of the scan SNR ([jhk]_snr) or the photometric uncertainty SNR (1.0857/[jhk]_cmsig). The initial reliability score is set to the maximum score in any detected band according to the following SNR thresholds that are derived from analysis of the curves in Figures 6-10.

2-Band Detections:

JH detections

rel         A       B       C       D       E       F
J: snr    >4.53   >4.18   >3.88   >3.50   >3.19   <3.19
   cmsig  <0.24   <0.26   <0.28   <0.31   <0.34   >0.34

H: snr    >4.18   >3.88   >3.62   >3.39   >3.19   <3.19
   cmsig  <0.26   <0.28   <0.30   <0.32   <0.34   >0.34


HKs detections 

rel         A       B       C       D       E       F
H: snr    >5.17   >4.52   >4.18   >3.88   >3.50   <3.50
   cmsig  <0.21   <0.24   <0.26   <0.28   <0.31   >0.31

Ks:snr    >5.17   >4.52   >4.18   >3.88   >3.50   <3.50
   cmsig  <0.21   <0.24   <0.26   <0.28   <0.31   >0.31

JKs detections

rel         A       B       C       D       E       F
J: snr    >5.43   >4.52   >4.18   >3.88   >3.50   <3.50
   cmsig  <0.20   <0.24   <0.26   <0.28   <0.31   >0.31

Ks:snr    >4.72   >4.34   >4.02   >3.74   >3.39   <3.39
   cmsig  <0.23   <0.25   <0.27   <0.29   <0.32   >0.32


1-Band Detections

rel         A       B       C       D       E       F
J: snr    >7.76   >6.03   >5.43   >4.94   >4.94   <4.94
   cmsig  <0.14   <0.18   <0.20   <0.22   <0.25   >0.25

H: snr     ---    >7.24   >6.39   >5.43   >4.34   <4.34  
   cmsig   ---    <0.15   <0.17   <0.20   <0.25   >0.25

Ks:snr    ---    >9.87   >7.24   >6.03   >4.94   <4.94
   cmsig   ---    <0.11   <0.15   <0.18   <0.22   >0.22

(Note - Best reliability score for H-only and Ks-only sources is "B")

Extractions with no valid detections

rel='F' if rd_flg NOT MATCHES '*[1,2,3]*'

ii. Final Score Revision

For 1- and 2-band detections that do not already have an initial reliability score of rel='F', set rel='F' if any of the following conditions are met. These conditions are derived from quality criteria developed as part of source selection for the All-Sky PSC:

chi-squared test (single-frame events - cosmic rays, meteor trails, hot pixels)

rd_flg NOT MATCHES '*[1,3]*' AND 
rd_flg[b] = '2' AND [jhk]_cmsig > 0.1 AND [jhk]_psfchi > 2 (in all detected bands)

Coverage test (low coverage noise events, scan edge effects)

rd_flg NOT MATCHES '*[1,3]*' AND
ndet[2,4,6] < 3 (in all detected bands)

Frame detection test (single-frame events - cosmic rays, meteor trails, hot pixels)

rd_flg NOT MATCHES '*[1,3]*' AND
(j_m < 14.5 and ndet[1]/ndet[2] < 0.41 (if J-band detection) AND
 h_m < 14.0 and ndet[3]/ndet[4] < 0.41 (if H-band detection) AND
 k_m < 13.5 and ndet[5]/ndet[6] < 0.41 (if Ks-band detection)

Note - The magnitude limits used for 6x point source WDB are: 
       15.9 (J), 14.8 (H), 14.4 (Ks) 
       This is the only magnitude-dependence used in the point
       source reliability assignment.

[Last Update: 2006 October 1, by R. Cutri]

Previous page. Next page.
Return to Explanatory Supplement TOC Page.


Figure 1 - Detection fraction versus magnitude by band for sources in the 90301 calibration field that was scanned a total of 3430 times.	Figure 2 - Detection frequency histograms for sources in the 90301 calibration field. J distributions are shown in blue, H in green and K_s in red.	Figure 3 - Close ups of the low and high detection frequency peaks in the distributions shown in Figure 2.


Figure 4 - Log histograms of the number of reliable (black curves) and unreliable extractions (red curves) as a function of magnitude for the Survey PSWDB truth table extractions located in the area covered by the 90266, 90272, 90301 and 90330 calibration fields.	Figure 5 - Log histograms of the number of reliable (black curves) and unreliable extractions (red curves) as a function of photometric measurement uncertainty for the Survey PSWDB truth table extractions in the 90266, 90272, 90301 and 90330 calibration field areas.	Figure 6 - Net differential reliability as a function of photometric measurement uncertainty for the Survey PSWDB "truth table" extractions in the 90266, 90272, 90301 and 90330 calibration field areas.


Figure 7 - 3-band extractions	Figure 8 - 2-band (JH) extractions	Figure 9 - 2-band (HK_s) extractions	Figure 10 - 2-band (JK_s) extractions	Figure 11 - Single-band extractions
Net reliability vs. photometric measurement uncertainty ([jhk]_cmsig) for survey PSWDB truth table extractions in all calibration tiles. Top panels shows J (blue), H (green) and K_s (red) reliability curves. Bottom three panels show xmsig histograms of reliable (black) and unreliable (magenta) extractions in J, H and K_s.