VI. Analysis of the Release Catalogs

5. Completeness and Reliability

b. Extended Source Catalog

ii. Reliability

The Level 1 Specifications call for 99% reliability at galactic latitude > 20° from the Galactic plane, and 80% reliability between 10° and 20°. The specification indicates that completeness and reliability be calculated "per magnitude in the faintest magnitude bin." 2MASS has used the Level 1 sensitivity specification (15.0, 14.3, and 13.5 mag for 10- at J, H, and K_s, respectively) to define the faint end of the lowest bin in each waveband.

The XSC consists of two subsets, Extended Sources and Galaxies. Each candidate source was given a numeric flag, rating how extended it was and how galaxy-like it was, and sources passing a given threshold on either parameter were placed in the Catalog.

Because the goal of the XSC was to produce a reliable galaxy catalog, the Level 1 Specifications were interpreted as the galaxy reliability of the sources that exceeded the threshold for the galaxy flag. This is much more stringent than simply requiring that a source be extended. Double stars below the resolution of 2MASS are correctly picked up as extended sources, but they count as false sources for the galaxy reliability, since they are not galaxies.

Reliability is usually the hardest specification to measure, primarily due to the large variations in conditions encountered in an all-sky catalog. That is, one can analyze a subset of the data to determine its reliability, but this gives no assurance that different reliability problems do not occur elsewhere.

However, nearly one-third of the entire 2MASS XSC has been reviewed source by source by eye (see IV.5c1), with each source verified against its 2MASS image and an optical DSS image, when available. As far as we know, this produces the best reliability determination for any such large catalog at the time of the Release.

For bright sources, this reliability determination is definitive, and the entire set of sources in the XSC is virtually 100% reliable, with only a few false sources, due to missed artifacts. However, for fainter sources, a few percent of the sources cannot be definitively classified as galaxies, double stars, or misclassified single stars. Fortunately, a statistical analysis (below) of these sources has found that at least 85% of them are most likely true galaxies. We use that minimum estimate of 85% to calculate our best estimate of the reliability below.

The reliability from visual inspection (see below) for all sources with g_score < 1.4 (i.e., the ones that passed the "galaxy" threshold) with |b| > 25°, is as follows:

Band N_gal N_not_gal N_unk Rel_min Rel_max Rel_est
J 59,999 19 1919 96.9% 100.0% 99.5%
H 94,284 212 2696 97.0% 99.8% 99.4%
K 93,030 259 3589 96.0% 99.7% 99.2%

Band	N_gal	N_not_gal	N_unk	Rel_min	Rel_max	Rel_est
J	59,999	19	1919	96.9%	100.0%	99.5%
H	94,284	212	2696	97.0%	99.8%	99.4%
K	93,030	259	3589	96.0%	99.7%	99.2%

As can be seen, it is quite likely that the XSC exceeds the stringent Level 1 specifications for galaxy reliability for |b| > 20°.

Figure 1 shows the reliability vs. magnitude, ignoring the unknowns.

Figure 1

In addition, we compared all sources with g_score = 1 against the SDSS EDR, finding a reliability ~99% at the Level 1 specifications. This comparison is shown in Figure 2.

Figure 2

(The dip in the 2MASS/SDSS match rate at 8.5 to 11 mag is apparently a feature in the SDSS. The unmatched XSC sources in this range are without question galaxies, with NED counterparts, and often with spiral structure visible in the 2MASS Atlas Images. In particular, T. Jarrett has looked at every single XSC source brighter than K_s = 12 mag, and nearly every one of them is clearly a galaxy. The reliability of this set from the visual analysis is 100.0%, to one decimal place, as shown in Figure 2.)

For galactic latitudes between 10 and 20°, we significantly exceed the Level 1 specification. From the visual inspection, the reliability is well above 90% in the appropriate magnitude bins, as shown in the first plot above.

A number of other tests have been performed, including:

Making a detailed mosaic of the Virgo Cluster using the 2MASS data, and extracting galaxies manually from it. See Virgo Cluster Mosaic Comparison.
Abell 3558 galaxy cluster repeatability from 15-30 repeated scans

All of these analyses verify the high reliability of the XSC.

An analysis of the calibration field repeats was done, but the number of galaxies in the field was too small to demonstrate anything, other than consistency with the above analyses. For additional analysis, see below.

The Nature of Galaxy Truth Table "Unknowns"

In order to verify the results of the 2MASS galaxy processing, over 150,000 sources were examined by hand and classified as galaxies, apparently single stars, double stars, triple stars, and artifacts. The parameters calculated by GALWORKS, the 2MASS Atlas Images, and the DPOSS images were all employed.

However, at high latitudes and in the half magnitude bin above the XSC completeness specification, 4--10% of all sources defied categorization in that they could not clearly be placed into either the galaxy or single-star classes. These sources were left as a class of "unknown" sources.

A few unknowns have been followed up with higher resolution data, and most of them have turned out to be true galaxies. However, the number of such followups is extremely small, and because all of them have occurred in known galaxy clusters, that subset of the unknown sources might not be representative of the class as a whole. In the absence of definitive information, the unknowns are considered as nearly always true galaxies, or that they were nearly always falsely-detected single stars.

Finally, enough statistics have been accumulated to state with some confidence that most of the unknowns that have J magnitudes between 14.5 and 15.0, found above |b| = 30°, are likely true galaxies. It is possible that a significant minority (~15%) could still be false sources.

In the following, the term single stars or just singles will always mean "detected extended sources" classified as single stars.

Since the unknowns are either true galaxies, falsely-detected single stars, or a mixture of the two classes, we can compare the characteristics of the unknowns to the two possible source classes. Thus, consider the following two hypotheses, and their implications:

The unknowns are true galaxies whose non-nuclear components have become too faint to be detected by eye on the 2MASS or DPOSS images, and thus appear quasi-stellar. In other words, the fuzz around the nucleus has become too faint to be seen by anything other than the power of the GALWORKS processing.
This hypothesis predicts that the characteristics of the unknowns are very similar to those of true galaxies, except that the unknowns should be fainter on average and have somewhat lower measured source extents, accounting for the less visible fuzz.
The unknowns are actually single stars which have been detected by GALWORKS for the same reasons that the sources classified as single stars were detected.
Unfortunately, we cannot predict from first principles the characteristics that this implies, since we have not yet analyzed the single star classifications to find the main failure modes that allowed these stars to be detected as extended. The most likely suspect is a brief instance of untracked seeing that was either too brief to be reliably detected by the automatic see_track "seeing-tracking" processing, or that was associated with a marginally-poor see_track score that fell just under the threshold for rejecting the data. Untracked seeing is a well-known cause that creates extended sources out of true point sources.
In spite of not knowing the precise cause that creates the category of single stars, this hypothesis predicts that the characteristics of the unknowns should closely match the characteristics of the single stars picked up as extended sources. There must be some modification to account for the placement of each source into the unknown category, rather than the single star category, but we cannot predict what those modifications would be, without further understanding of the failure mode that produces the single stars. For example, if the single stars are due to untracked seeing, then one scenario consistent with this hypothesis is that the unknowns result from instances of worse seeing than the occasions when a single star category was clear. The worse seeing would mimic a fuzzier source, preventing a confident assignment of the source to the single star category.

With these two hypotheses in mind, it is now a simple matter to analyze all the main characteristics of these source populations and compare them to the predictions of each hypothesis. Each characteristic will be considered in turn.

In all analyses below, we consider the set of classified sources with 14.5 < J(mag) < 15.0 that are unconfused as indicated by their cc_flg, found at |b| > 30°. There are 3467 galaxies, 41 singles, 25 doubles, 0 triples and 178 unknowns. (See below.) Due to the large difference in the number of galaxies, singles and unknowns, in some of the figures below the number of galaxies and singles have been scaled to the number of unknowns, to make the comparison of the distributions clearer.

Galaxies have an approximately uniform distribution on the sky, whereas stars are highly concentrated to the Galactic plane. If the unknowns follow one distribution, rather than the other, this test by itself is probably the most powerful test by far, giving the answer directly.

The distribution of stars falsely identified as extended by untracked seeing will not be precisely the same as the stellar distribution, since untracked seeing is somewhat more likely to occur in low source density areas, where the seeing is tracked with a lower frequency. However, seeing "flare-ups" do occur on time scales too short to be tracked even with the high stellar densities of the Galactic plane. In any case, the theoretical distribution is not needed, since the observed distribution of the single stars is available.

Because the area covered by this subset of the "truth table" is not readily available, we instead simply compare the histogram of number of sources vs. density, in Figure 3. (Density is the log number of sources with K_s < 14.0 mag.)

Figure 3

The histogram of unknowns vs. density is almost an exact match of the histogram of galaxies vs. density below density = 2.9, and definitely disagrees with the histogram of singles vs. density below density = 2.9. This is the most powerful evidence that most of the unknowns are, in fact, true galaxies.

Above density = 2.9 there is an excess of unknown sources, implying that a small percentage of unknowns may, in fact, be false sources in those high density areas. There are 27 more unknowns above density = 2.9 than predicted by the hypothesis that 4.4% of all galaxies fall into the unknown category. This implies that ~27/178 = 15% of the J unknowns are singles, and that ~85% of the J unknowns are true galaxies.

In the figures below, the unknowns will be separated into sources found in areas with density < 2.9 and with density > 2.9.

Galaxies have a range of measured extent parameters, ranging from small compact galaxies to large more diffuse galaxies. The hypothesis that unknowns are galaxies whose non-nuclear component has fallen below some visibility threshold implies that the unknowns should continue to show a variation in their extent parameters, but that the group of unknowns as a whole must be fainter (see below) and therefore have smaller extent parameters.

If these predictions are not obvious, consider two specific cases: First, a small compact galaxy whose fuzz is bright enough to be seen on the 2MASS Images, and therefore is classified as a galaxy. If that galaxy is simply farther away, it becomes smaller and fainter, with a smaller measured extent. Correspondingly, it gets harder to classify it as a galaxy. Second, consider a large diffuse galaxy, also classified as a galaxy. If that galaxy is farther away, it also becomes smaller and fainter, with a smaller measured extent. However, this galaxy still has a large measured extent, compared to the first galaxy, even though it too has become too faint to classify as a galaxy. Hence, the group of galaxies as a whole has become fainter and smaller, but still retains some variation in the measured extent parameters.

The J shape score (j_sh_sc) is the measured source extent divided by the one scatter in the shapes of point sources. Comparing that score vs. J magnitude directly shows the range in measured galaxy source size at a given total J magnitude, as well as the general decrease in measured source extent, as the total J magnitude becomes fainter.

The relation for unknowns (Figure 4) is exactly that predicted for galaxies (Figure 5), and very different from that for singles (Figure 6). Only one single has a j_sh_sc > 20, whereas a considerable number of the unknowns have scores above 20. Further, the unknowns show a clear trend of declining score with fainter magnitude, not present in the singles.

Figure 4 Figure 5 Figure 6

We conclude that we have strong confirming evidence that the bulk of the unknowns are true galaxies.

The distribution of the J and K_s measured shapes are significantly different for galaxies (Figure 7) and singles (Figure 8). Galaxies show a high correlation between their shapes measured at J and K_s, whereas the singles considered here, detected at J, tend to not have very high K_s shapes. This latter fact may be caused by the J and K_s fitted seeing curves differing significantly in areas with untracked seeing.

Figure 7 Figure 8 Figure 9

Regardless of the source of the difference between galaxies and singles, there is no doubt that the unknowns (Figure 9) strongly resemble galaxies, and not singles.

For completeness, we present other possible comparisons that use the 2MASS data, but which are not expected on theoretical grounds to show clear differences between galaxies and stars.

The color-color diagrams for galaxies (Figure 10), singles (Figure 11) and unknowns (Figure 12) closely resemble each other, due to the color selection used to assign the g_score. Nonetheless, it is noticeable that the singles include relatively more blue sources (J-H < 0.4) than are seen in the galaxy and unknown plots.

Figure 10 Figure 11 Figure 12

A histogram of the number as a function of magnitude (Figure 13) for the three source classes is difficult to interpret, since there are many effects at work. For example, it is expected that the unknowns would show an increased number at fainter magnitudes if they are galaxies, due to the selection effect of being classified as unknowns. It is also expected that stars falsely classified as extended would increase rapidly at fainter magnitudes, due to the larger errors in derived parameters at fainter magnitudes. Thus, we can draw no firm conclusions from this figure.

Figure 13

Three powerful comparisons strongly support the hypothesis that the unknowns are largely (but not necessarily entirely) galaxies and strongly reject the hypothesis that the unknowns are single stars. The small differences in the distribution of parameters for unknowns vs. galaxies are exactly those predicted by the hypothesis that the unknowns are simply galaxies that have become faint enough, so that their "fuzz" is not clearly distinguishable in human analysis of images.

These comparisons leave little doubt that the unknowns are largely galaxies. Additional confirming evidence may remove any doubt once we understand the origin of the singles. For example, if the singles are due to untracked seeing in one way or another, that gives another distribution to which the unknowns and galaxies can be compared.

Of course, the ultimate truth is further followup of a number of the unknowns. The very limited work to date has found that the unknowns are indeed galaxies, but more followup is needed. Again, it is possible that ~15% of the unknowns are false detections of single stars. Once the origin of the singles is known, it may be possible to accurately determine the percentage of the unknowns that are galaxies and the percentage that are single stars.

Further Analysis of Reliability

The XSC is first and foremost a catalog of resolved objects, the bulk of which are galaxies in the local Universe (redshift < 0.3). A very small percentage are Milky Way stars (point sources), masquerading as resolved, extended objects. The contamination level rises with the confusion noise. The degree of stellar confusion is measured by the density metric (see IV.5c).

The point source contamination or reliability of the XSC may be gauged using repeated observations of the Abell 3558 cluster of galaxies (located behind a region of the Milky Way rich in stars). The results are documented in Galaxy Cluster Repeatibility Analysis. Results are also shown in Figure 14.

Figure 14

Approximately one-third of the XSC has been visually inspected (see IV.5c1), with nearly 100% completeness at the bright end: K_s < 12.5 mag. The results of this inspection then represent a measure of the reliability of the Catalog, particularly at the bright end, since it was uniformly inspected. The results are divided into four samples, each representing a "density" or confusion noise regime:

low stellar density: |glat| > 25°
moderate stellar density: 10° < |glat| 25°
high stellar density: 5° < |glat| < 10°
Milky Way: |glat| < 5°

The density metric and its relation to the Galactic Plane are explained in IV.5c1. The "raw," or aggregate, reliability is summarized in Figure 15.

Figure 15

The bottom line is that the XSC reliablity is better than 98% for unconfused regions of the sky, and better than 90% (K_s < 13.0 mag) for regions that include the Milky Way. The tabular results for the low stellar density case are summarized here:

|  mag | Ngal| Nbog|  Rj | Nunk|  Ngal| Nbog|  Rh | Nunk|  Ngal| Nbog|  Rk | Nunk|
| (1)  | (2) | (3) | (4) | (5) | (6)  | (7)  | (8)|(9)  | (10) | (11)|(12) | (13)|
  9.000   985     0 100.0     4   2109     0 100.0    10   2994     0 100.0    17
 10.500  2016     0 100.0     5   5207     1 100.0    29   8090     2 100.0    29
 11.250  2512     0 100.0    19   6927     0 100.0    17  10761     0 100.0    55
 11.750  5178     1 100.0    14  13247     1 100.0    72  21540     2 100.0   244
 12.250  9867     1 100.0    40  26045     1 100.0   395  44374     0 100.0  1214
 12.750 19147     0 100.0   197  50682     0 100.0  1164  61565    64  99.9  1215
 13.250 37481     1 100.0  1159  51421    86  99.8  1231  32297   328  99.0  2575
 13.750 58739    31  99.9   854  28617   332  98.9  2461  23769   563  97.7  6425
 14.250 37439   295  99.2  2008  21318   556  97.5  6142  14581   677  95.6  6211
 14.750 23068   473  98.0  4226  11526   411  96.6  4978    801    46  94.6   437
 15.250 18368   688  96.4  7076    504    17  96.7   397     27     2  93.1    28
 15.750  5217   125  97.7  2548     15     1  93.8    18      6     0     -     1

Notes:
Statistics are for low density regions of the sky: |glat| > 25 deg

column (1) is the elliptical isophotal mag
columns (2)-(5) are the J-band results, with (2) number of extended sources, (3) number of false
        sources, (4) %Reliability, and (5) the number of 'unknown' sources
columns (6-9) are the H-band results; see above
columns (10)-13) are the K-band results; see above

The Level-1 Science Requirements set specific completeness and reliability requirements. Here the requirements allow the application of the g_score, to improve the galaxy reliability (representing most of the XSC, in any case). The bottom line is that the XSC galaxy reliablity is better than 99% for unconfused regions of the sky, and better than 90% (K_s < 13.0 mag) for regions that include the Milky Way. The tabular results for the low stellar density case are summarized here:

|  mag | Ngal| Nbog|  Rj | Nunk|  Ngal| Nbog|  Rh | Nunk|  Ngal| Nbog|  Rk | Nunk|
| (1)  | (2) | (3) | (4) | (5) | (6)  | (7)  | (8)|(9)  | (10) | (11)|(12) | (13)|
  9.000   595     0 100.0     4   1628     0 100.0     9   2457     0 100.0    15
 10.500  1882     0 100.0     5   5030     1 100.0    27   7865     1 100.0    26
 11.250  2434     0 100.0    18   6806     0 100.0    15  10616     0 100.0    44
 11.750  5063     1 100.0    14  13092     1 100.0    70  21315     1 100.0   235
 12.250  9742     1 100.0    34  25770     0 100.0   382  43958     0 100.0  1168
 12.750 18944     0 100.0   190  50281     0 100.0  1117  61095    42  99.9  1137
 13.250 37151     0 100.0  1134  51089    61  99.9  1180  31935   217  99.3  2452
 13.750 58368    19 100.0   785  28304   238  99.2  2364  22950   348  98.5  5924
 14.250 37213   220  99.4  1940  20698   344  98.4  5801  13372   402  97.1  5464
 14.750 22786   338  98.5  4116  10608   248  97.7  4368    729    31  95.9   383
 15.250 17650   417  97.7  6579    439    10  97.8   336     26     1  96.3    24
 15.750  4485    46  99.0  2044     14     1  93.3    14      5     0 100.0     1

[Last Updated: 2003 Jan 31; by T. Jarrett and T. Chester]

Return to VI.5.