INTRODUCTION
In order to assess the repeatability of the results of active deblending using PROPHOT, the algorithm has been run on a series of 12 repeated scans of the same region, and the results will be presented below. The goals of the tests were:
(1) Assess the repeatability of the results for different scans.
(2) Determine the errors in flux estimation by comparing the different estimates of the same blend.
The repeatability results provide some useful information on completeness and reliability, which may be defined as follows:
Completeness = the probability that a genuine blend will be reported
Reliability = the probability that a reported blend is genuine
Clearly the completeness must depend on the physical parameters of the blend, which may be characterized by the primary magnitude, the magnitude difference of the components, and the separation. Thus the completeness is meaningful only in the context of specified ranges of these parameters.
In order to assess completeness and reliability, we need some criteria for recognizing a true blend. In the analysis which follows, we will assume that a blend is genuine if either one of the following statements is true:
(1) The blend is detected in more than 1 scan
(2) The blend is detected in more than 1 band in the same scan.
These criteria are, of course, not foolproof -- there will be occasions whereby a false blend will satisfy one or both conditions through random chance. In the analysis which follows, we will try to assess the probability of being misled by the coincidence of false blends in separate scans and separate bands.
An additional consideration for the assessment of completeness is that we need some knowledge of how many blends there are in the region of the sky being observed. Since a truth table is not available, we need to make some further assumptions. Specifically, we will assume that, during the course of 12 scans at 3 bands, all blends within the survey sensitivity limits will have been seen at least once. In this context, "seen" means that the blend has satisfied the above criteria for being classified as genuine. Thus, in effect, we are assessing the completeness of a single scan based on the results of a set of scans.
Based on the above assumptions, the completeness, C, and reliability, R, can be related to the quantities derived from the observed data by:
C = Ngen/(12 * Nblends)
R = Ngen/Ncan
where Ncan is the total number of candidate blends in the source files from all 12 scans, Ngen is the number of those candidates which have been classified as genuine on the basis of the above criteria, and Nblends is the number of spatially-distinct genuine blends inferred to be in the region itself.
The Nblends parameter actually represents a lower limit to the true number of
blends present in the region since some blends will undoubtedly be missed,
even after 12 scans. Thus the completeness, C, as defined above, represents
only an upper limit to the true completeness. Therefore, C
should more correctly be termed the "repeatability" rather
than the "completeness". We will, however, continue using the term
"completeness" since it is closely related to the true completeness, and
has more intuitive appeal than "repeatability".
DATA
The data selected for these tests consisted of 12 cal scans of a region in
the vicinity of zero galactic latitude. The specific scans were:
000325n/s127
000518n/s069
000518n/s096
000519n/s110
000613n/s095
000615n/s112
980830s/s011
980830s/s068
990503s/s107
990916s/s012
000421s/s076
000530s/s102
The seeing shape parameter for these scans ranged from 0.934 to 1.088.
ANALYSIS PROCEDURE
Active deblending in PROPHOT was carried out using a cutoff value of 1.3 for the reduced chi squared, i.e., all single-source solutions whose reduced chi squared exceeded this value were repeated using active deblending. The maximum allowed number of actively-deblended components was 2.
PROPHOT was run as part of the standard pipeline, culminating in the bandmerging process. The output source files (*.bfpts) contained typically 10,000 sources, 40% of which were members of active deblends, i.e., there were typically 2000 blended pairs per source file.
The matching from one scan to the next was done on the basis of a 9-sigma tolerance with regard to both position and magnitude, where sigma refers to the theoretical a-posteriori standard deviation calculated by PROPHOT.
As a sanity check on criterion #1 for recognizing a genuine blend, a source file of fake blends was created by taking each of the sources brighter than 15th magnitude in scan 1 and adding a fake companion of the same magnitude at a separation of 1 pixel. This file was then run through the matching procedure together with the .bfpts file from scan 2. The result was that only 2 matches were found out of a total of 5055 fake blends, i.e., the probability that we would mistakenly classify a false blend as genuine by criterion #1 would be only 0.0004.
After obtaining the scan-to-scan matches, the completeness and reliability
of the blends
were evaluated as a function of primary magnitude using the above expressions.
In addition, the magnitude errors of the primary and secondary source
components were estimated from the scatter of the estimates
from the different scans.
RESULTS
The completeness and reliability results for all detected blends (with no restrictions on flux ratio or separation) are presented in Table 1 as follows:
Pri mag Completeness(%) Reliability(%)
J H K J H K
9 ... 40.9 41.7 ... 98.2 100.0
10 43.1 41.2 34.5 100.0 97.6 92.3
11 41.9 37.5 41.1 95.1 97.9 93.4
12 40.6 42.4 41.1 95.7 96.4 95.6
13 41.9 39.2 36.7 95.6 97.4 96.7
14 40.6 36.5 29.9 95.0 96.4 95.3
15 35.1 26.5 13.6 92.9 90.4 80.5
16 25.2 12.0 ... 81.6 50.0 ...
The errors in the estimated magnitudes of the components are given in Table 2:
Mag Primary mag error Secondary mag error
(pri or sec) J H K J H K
9 ... 0.05 0.04 ... ... 0.32
10 0.03 0.04 0.24 ... 0.09 0.32
11 0.04 0.04 0.08 0.16 0.12 0.10
12 0.04 0.16 0.15 0.11 0.06 0.20
13 0.26 0.18 0.25 0.16 0.18 0.19
14 0.24 0.20 0.14 0.19 0.15 0.15
15 0.26 0.22 0.14 0.16 0.15 0.20
16 0.21 0.25 ... 0.18 0.61 1.96
The completeness and reliability in Table 1 were evaluated on the assumption that a detected blend is genuine if it shows up in more than one scan OR if it shows up in more than one band in a single scan. The validity of the former condition has been substantiated by the test with fake blends, described in the previous section. In order to investigate the validity of the second condition, the calculations were repeated using the more restricted definition whereby a blend is classified as genuine ONLY if it appears in more than one scan. The results are presented in Table 3, as follows:
Pri mag Completeness(%) Reliability(%)
J H K J H K
9 ... 40.9 44.4 ... 98.2 98.5
10 43.1 43.3 37.2 100.0 96.4 90.3
11 44.7 40.3 42.3 93.7 96.1 92.7
12 42.8 44.8 42.6 94.4 95.1 94.7
13 43.9 41.0 38.1 94.6 96.2 95.7
14 42.0 37.7 30.6 94.3 95.5 94.5
15 36.2 27.4 14.0 92.0 89.1 77.8
16 26.1 12.0 ... 80.3 50.0 ...
The fact that the results are not significantly affected by the imposition of the more conservative criterion for identifying genuine blends suggests that blends which are detected in more than one band tend to be also detected in more than one scan. In fact, this can be checked on the basis of the data themselves, whereby it is found that:
(a) 88% of sources which appear in more than 1 band in a given scan also appear in more than 1 scan
(b) 96% of sources which appear in all 3 bands in a given scan also appear
in more than 1 scan
The above calculations were repeated using restrictions on the ranges of source separation and pri-sec magnitude difference, whereby it was found that there was a relatively weak dependence on separation but a substantial dependence on magnitude difference. The following table (Table 4) shows the results for the case in which the minimum allowed separation is 1.5 arcsec, and the maximum allowed magnitude difference of the components is 2 magnitudes.
Pri mag Completeness(%) Reliability(%)
J H K J H K
10 47.2 55.3 46.2 100.0 100.0 94.1
11 44.0 48.4 55.1 100.0 100.0 99.0
12 51.4 53.2 51.0 99.2 98.3 98.2
13 51.2 44.1 39.9 98.6 98.7 97.9
14 45.2 37.4 30.4 97.7 97.1 95.9
15 35.8 26.9 13.8 94.1 90.9 80.5
16 25.6 12.0 ... 82.4 52.0 ...
A comparison of Tables 1 and 4 shows that the restriction on the range
blend parameters has resulted in significant increases in both
completeness and reliability.
DISCUSSION
The above tests have shown that the active deblending algorithm, currently implemented in the development version of PROPHOT, performs in a repeatable fashion with high reliability. For blends with separations of 1.5 arcsec or greater, primary magnitudes of 13 or less, and primary-secondary magnitude differences of 2 or less, the completeness (or, more correctly, scan-to-scan repeatability) is approximately 50% and the reliability typically exceeds 98%. Typical magnitude errors for the brighter sources are 0.04 for the primary and 0.1 or so for the secondary.
It is possible that the completeness would increase if we chose a different
criterion for the cutoff in reduced chi squared. The current value of 1.3
was chosen based on recent analyses of the distribution of reduced chi squared
values from previous PROPHOT runs. However, in that analysis, the reduced chi
squared values were adjusted in an attempt to correct for deficiencies in the
PROPHOT noise model caused by incorrect PSF variance maps; specifically, they
were rescaled so as to peak at unity. However, in the
PROPHOT runs used for the current tests, no such scaling was applied.
Since the noise-model errors tend to produce reduced chi squared values
substantially below unity,
it may well be that a cutoff of 1.3 would miss a significant
number of blends. It would therefore be interesting to repeat these tests with
a different chi squared cutoff. A more efficient alternative, however,
would be to wait until a new set of PSFs (with improved variance maps) has
been generated, and then rerun the repeatability tests using the same
chi squared cutoff of 1.3.
UPDATES:
Sept 28: RESULTS AS A FUNCTION OF SEPARATION AND FLUX RATIO
The repeatability tests were rerun, and the results expressed in terms
of source separation and pri-sec magnitude difference. For the purpose
of these tests, an upper magnitude limit of 14 was imposed on all components
at all bands. The results as a function of source separation were obtained
using an upper cutoff of 2.0 for the primary-secondary magnitude difference,
and are presented in the following table:
Sep["] Completeness(%) Reliability(%)
J H K J H K
1.00 ... 18.3 17.9 ... 78.6 65.2
1.50 27.1 16.7 22.1 92.9 75.0 50.0
2.00 29.9 27.1 28.5 95.6 94.9 86.8
2.50 48.6 37.0 36.6 99.4 97.1 92.2
3.00 60.7 36.5 36.1 99.5 96.2 92.9
Sep["] Primary flux error Secondary flux error
J H K J H K
1.00 ... 0.28 0.27 ... 0.34 0.25
1.50 0.11 0.23 0.35 0.31 0.60 0.51
2.00 0.10 0.17 0.13 0.18 0.15 0.22
2.50 0.04 0.04 0.08 0.09 0.10 0.16
3.00 0.04 0.04 0.04 0.08 0.15 0.08
In this table, the source separations have been grouped into 0.5" bins, whose center values are given in the "Sep" column. It is apparent that the various quantities listed in the table behave as expected, i.e. as the source separation increases, the completeness increases and the magnitude errors decrease.
Rebinning in terms of primary-secondary magnitude difference (denoted by Delmag), using a minimum source separation of 1.5 arcsec, we obtain:
Delmag Completeness(%) Reliability(%)
J H K J H K
0.00 38.7 28.2 30.9 98.8 93.1 92.3
0.50 46.0 24.7 27.5 97.2 95.2 83.1
1.00 51.2 36.2 36.9 99.1 94.7 89.6
1.50 41.7 40.6 34.3 100.0 98.6 90.5
2.00 50.0 33.8 32.1 94.1 94.4 90.1
2.50 36.7 25.7 22.5 95.7 94.7 72.9
3.00 37.5 31.2 27.3 96.4 96.8 87.8
Delmag Primary flux error Secondary flux error
J H K J H K
0.00 0.07 0.10 0.07 0.13 0.11 0.07
0.50 0.06 0.09 0.10 0.08 0.10 0.14
1.00 0.04 0.10 0.06 0.11 0.08 0.09
1.50 0.03 0.04 0.07 0.09 0.08 0.15
2.00 0.03 0.07 0.08 0.09 0.21 0.14
2.50 0.04 0.04 0.05 0.11 0.20 0.13
3.00 0.03 0.04 0.22 0.45 0.17 0.46
Again, the results show the expected behavior in that the primary magnitude
error decreases while the secondary magnitude error increases with increasing
Delmag.
If a double source is treated as a single source during parameter estimation
by PSF profile fitting, one would expect the flux to be underestimated.
To investigate the severity of the problem, a comparison was made between
the photometry results for the N=1 and N=2 cases (i.e. "no deblending" versus
"N=2 deblending"). The data used for this comparison consisted of three
of the cal scans used in the repeatability tests, specifically:
Each of these scans was run through PROPHOT using exactly the same parameters
as for the active deblending runs, except that N was set at 1, i.e. each blend
was fit as a single source. The procedure was then to take each blend from the
actively-deblended output and calculate the magnitude of the combined blend,
Mblend, and then obtain the magnitude bias, defined as (Msingle - Mblend), where
Msingle is the magnitude from the corresponding single-source fit. The
mean value of the magnitude bias was then determined for each band and each
scan, together with the standard deviation of an individual case.
Scatter plots of (Msingle - Mblend) v. Msingle for all 3 scans combined
are shown in
Figure 1 (J band),
Figure 2 (H band), and
Figure 3 (K band). The renegade points
(particularly at 14th magnitude and above) are presumably due to
misidentifications between the "blend" and "no blend" source files.
The means and standard deviations for all blends brighter than 14th magnitude
are presented in the following table:
If we compare the biases with the actual photometric errors obtained from
the repeatability tests (some of which were as low as 0.04 mag), it is apparent
that the biases are all statistically significant. The standard deviations
(denoted by sig in the above table) therefore reflect the range of magnitude
biases caused by the dispersion in such
variables such as the PSF width and the blend parameters.
Since the bias in an individual case can be expected to depend on the blend
parameters (particularly the separation and pri-sec magnitude difference),
the above analysis was repeated for restricted ranges of these parameters.
The results for the case of separations larger than 1.5" and magnitude
differences less than 1.0 were:
The biases show the expected behavior in that they become more severe in
the presence of blends whose components have comparable magnitudes and
substantial separation.
In order to show more clearly the effect of pri-sec magnitude difference
and source separation
on the single-source magnitude bias, scatter plots have been made of
(Msingle - Mblend) v. (pri-sec magnitude difference)
and (Msingle - Mblend) v. separation, for a magnitude range
Mblend <= 13. The results are as follows:
(a) Flux bias v. pri-sec magnitude difference:
(b) Flux bias v. source separation:
The trend of increasing bias with decreasing pri-sec magnitude difference
is quite clear. For H and K bands in particular, the flux bias is
approximately 0.5 mag when the components are of comparable magnitude.
BOTTOM LINE: If we don't deblend, then the "single-source" photometry
can easily be off by a few tenths of a magnitude.
The position repeatabiliity of active deblending was investigated using a
set of partially overlapping survey scans. The scans used for this purpose
were from 980319s, in the overlap regions of the scan pairs as follows:
Sept 29: THE PENALTY SUFFERED BY NOT DEBLENDING
Scan 1: 000325n/s127
Scan 2: 000518n/s069
Scan 3: 980830s/s011
Scan # J bias sig H bias sig K bias sig
1 0.18 0.14 0.33 0.18 0.32 0.19
2 0.29 0.16 0.38 0.18 0.34 0.20
3 0.22 0.16 0.38 0.20 0.36 0.21
Scan # J bias sig H bias sig K bias sig
1 0.34 0.17 0.45 0.16 0.43 0.18
2 0.40 0.23 0.49 0.18 0.42 0.24
3 0.37 0.20 0.50 0.20 0.45 0.23
Figure 1a (J band)
Figure 2a (H band)
Figure 3a (K band).
Figure 1b (J band)
Figure 2b (H band)
Figure 3b (K band).
Oct 2: PLOTS OF PHOTOMETRIC ERRORS FROM ACTIVE DEBLENDING
J, H, and K plots of the RMS magnitude as a function of mean magnitude
derived from the repeatability tests are shown in
Figure 4. Values for the primary and secondary
components are plotted with "+" and "." symbols, respectively.
Data corrresponding to component separations of less than 1.5 arcsec were
excluded from the plot.
Apr 26: POSITION REPEATABILITY
22 & 23
24 & 25
25 & 26
26 & 27
35 & 36
37 & 38
38 & 39
41 & 42
There was a total of 203 active deblends in the overlap regions. In each case,
an attempted match was made between the source lists for the scan pair, and
the position residuals were calculated for the successful matches. The
RMS value of the position residuals were calculated, expressed in both
arcsec and sigmas, where sigma is the theoretical position error. These
RMS values were binned as a function of magnitude, delta-magnitude (i.e.,
the magnitude difference between primary and secondary), and component
separation. The results were as follows:
J Magnitude
10.000 11.000 12.000 13.000 14.000 15.000
________________________________________________
Pri[arcsec] | ... ... 0.086 0.102 0.186 0.230
Sec[arcsec] | ... ... 0.106 0.124 0.158 0.188
|
Pri[sigmas] | ... ... 0.607 0.704 1.052 1.317
Sec[sigmas] | ... ... 0.599 0.880 1.078 1.240
J Delta-magnitude
0.000 0.500 1.000 1.500 2.000 2.500
________________________________________________
Pri[arcsec] | 0.143 0.151 0.166 0.184 0.339 0.155
Sec[arcsec] | 0.216 0.170 0.166 0.139 0.229 0.097
|
Pri[sigmas] | 0.916 1.004 1.043 1.151 1.489 1.046
Sec[sigmas] | 1.450 1.124 1.142 0.893 1.484 0.641
Separation [arcsec]
1.500 2.000 2.500 3.000 3.500 4.000
________________________________________________
Pri[arcsec] | ... 0.159 0.192 0.181 0.226 0.068
Sec[arcsec] | ... 0.183 0.171 0.167 0.170 0.134
|
Pri[sigmas] | ... 0.959 1.052 1.123 1.266 0.536
Sec[sigmas] | ... 1.244 1.146 1.040 1.196 0.884
Overall RMS position error [sigmas] = 1.25
It is apparent that the RMS position errors are completely consistent with
the error bars quoted in the PROPHOT output source files.