Bandmerge is a software package developed at the Spitzer Science Center. It is designed to take in ASCII tables of positions and fluxes of detected astronomical sources in 2-7 different wavebands, and write out a single table of the merged data. The tool was designed to work with source lists generated by the Spitzer Science Center’s MOPEX software, although it can be “fooled” into running on other data as well.
This manual describes the software and interface in full. Bandmerge is compatible with Solaris, MacOSX and most flavors of Linux, although we only provide full user support for Linux RedHat.
1.1 Important Documentation
The following documents and webpages are recommended reading, and are referred to throughout this User’s Guide:
The Spitzer Helpdesk also contains a KnowledgeBase of frequently asked questions.
If you still have questions about Bandmerge then you can get help by submitting a ticket to the Helpdesk. Please help us by including the following information:
The operating system you are running (e.g. Mac Intel OS10.5/Linux Redhat/Solaris 10);
Which version of Bandmerge you’re using;
The AOR ID of your data (if Spitzer data);
Attach your namelist;
A description of the error message you saw;
As much information as possible about the problem, including any processing steps you carried out since downloading the data from the archive.
Please note that our system does not autoreply, so if you receive a reply from us within a few minutes then please check it for further information.
1.3 Overview of Bandmerge
Bandmerge was written as a tool to combine source lists of the same field, at different wavelengths, into a single merged data table. Spitzer users frequently observe their target fields at more than one wavelength but, since each channel is reduced separately, this results in a target list for each channel, almost always with a different number of detected sources in each one. Bandmerge reads in these source lists, matches the positions of the detected sources in each of them, and writes out a single merged table with the source position and flux at each wavelength. Because images at different wavelengths can have very different pixel scales, Bandmerge reads in source lists in the galactic coordinates and projects the source positional (RA, Dec) information in the 7 different bands onto a common 2-dimensional plane with corresponding (x,y) positions. The bandmerging is performed in the fiducial plane in (x,y) positions, and the merged catalog with the (x,y) positions is then converted back to the galactic coordinates during the final output. When performing positional matching, Bandmerge takes into account the positional uncertainties, and more importantly, it can derive the systematic offsets between two different bands, correct this offset and perform more accurate source matching.
Once you have downloaded and unpacked the Bandmerge software, you need to set it up to run on your system. In the Bandmerge installation directory, you will find the file bandmerge.csh. You need to edit the first line of this file to match your installation directory as follows:
Bandmerge can be run from any directory, but wherever you choose to run it from, there must be a subdirectory called cdf/. This subdirectory must contain the input parameter files required to run Bandmerge (see §Chapter 4).
1.5 User Interface
Bandmerge is run on the command-line by calling a single Perl wrapper script called bandmerge.pl, and uses an input parameter “namelist” (*.nl) file to control the input and output files and parameters. A detailed description of the Perl script can be found in the file README_bandmerge, which is distributed with the software. The namelist file is a simple ASCII text file, described in §2.2. An example can be found in Appendix A the cdf/ directory of your Bandmerge installation.
The syntax for running Bandmerge is as follows:
unix% bandmerge.pl –n bandmerge.nl
where the bandmerge.nl parameter namelist file must be located in a subdirectory cdf/ of the current working directory where you are running Bandmerge (<working_dir>). Table 1.1 lists all of the files that must be present for Bandmerge to run. See §3 for a description of all the input files. If the full path to the input files is specified in the namelist then the <data_dir> can be anywhere. If the path is not specified then <data_dir> must be the same as <working_dir>.
Table 1.1: Required input files, with examples and locations
The first bandmerging phase is cross-band linkage, in which every qualifying detection has an opportunity to select up to three matching qualifying detections from each band other than its own. Initial matching is done on a position-only basis; a coarse spatial window is used to select possible matches for each seed detection, and any candidates in the coarse window are subjected to the fine position test, which requires that the position-discrepancy chi-square have a value below a user-controlled threshold. If more than one candidate passes the fine position test, an alternative figure of merit called the pseudo-chi-square is evaluated, and used instead of the position chi-square value. The pseudo chi-square parameter is intended to have the same property as the chi-square in that larger values indicate less desirable matches. The computational definition of the pseudo-chi-square is currently not implemented, and the code uses the position chi-square as the pseudo-chi-square value. In each candidate band, up to three candidate detections with the lowest pseudo-chi-square values are kept for the seed by storing pointers to these detections. A counter also maintains the information about the total number of candidates that passed the match tests, and this is used in the confusion-flag information in the output file. All bands are processed as seed bands. When cross-band linkage is finished, every detection has pointers to any acceptable matches in all other bands, up to three per band.
1.6.2 Inconsistent Chain Processing
Ideally, cross-band linkage would end such that for any given detection there would be at most one pointer to another detection in each other band, with the source being pointed to having a reciprocal pointer back to the given detection. This is the most common case, in fact. But close sources, especially in regions of high density (typically low Galactic latitudes), often get tangled up with each other because of position estimation errors. In such cases, inconsistent chains can develop. These are chains in which a given detection’s preferred match in a different band points to a different source in the given detection’s band. Figure 1.1 shows an example of an inconsistent chain in band B1 (with sources S1, S3 and S5) and band B2 (with sources S2, S4 and S6). Because the match-quality parameters are required to be symmetric under interchange of seed and candidate, such chains cannot be closed loops; instead they must terminate in a reciprocal relationship. The program follows such chains until it finds the reciprocal relationship. Then breaks the chain at the point just prior to this relationship. Wherever a linkage is broken, any existing scond-choice pointer is elevated to first-choice status, and any existing third-choice pointer is elevated to second-choice status. This processing is done for all pointer linkages until only reciprocal linkages remain.
Figure 1.1: An Inconsistent Chain
1.6.3 Excess Linkage Processing
Even after all linkages have been made reciprocal, it is still possible for a different kind of inconsistency to exist in the chain. For example, a detection in one band may have a reciprocal relationship with a detection in band two, which has a reciprocal relationship with a detection in band three, which has a reciprocal relationship in band one that is not the detection in band one with which we started the chain. Thus two band-one detections are in the chain; this is called excess linkage. Figure 1.2 shows an example of excess linkage with two sources (S1 and S4) in band one, one source (S2) in band two, and one source (S3) in band 3. One of the two band-one detections must be removed. This is done by breaking the weakest link, i.e. the link with the largest pseudo-chi-square value. This processing is done until all chains have no excess linkages. Unlike inconsistent chain processing, breaking a link does not involve elevating second-choice or third-choice pointers, because this would necessitate a complete reprocessing for inconsistent chains.
Figure 1.2: Excess Linkage
1.6.4 Linkage Rejection Processing
After excess-linkage processing, only one kind of problem can remain: not all detections in the chain are linked to each other. A simple example is: a band-one detection is linked to a band-two detection, and the latter is linked to a band-three detection; but the band-one and band-three detections are not linked to each other (see Figure 1.3; left). This is referred to as linkage rejection; at least one detection is linked to two others who have rejected each other. Since the mutually-rejecting detections are indirectly linked through their common detection(s), there is some indication that perhaps they should be allowed to link to each other, and so a looser threshold is applied. This involves scaling the threshold by the user-controlled parameter LR_Scale_Fact with default value = 1.5. If confusion has been diagnosed then the pseudo-chi-square threshold is scaled by this factor, otherwise the unconfused threshold is scaled. If this fails, the entire chain is searched for the detection with the fewest links and the link(s) to this source is/are broken. If more than one detection has the fewest links, then the link to the detection with the weakest link(s) (even for detections not involving a band with more than one detection in the chain) is/are broken. This process is illustrated in Figure 1.3 (right). The sources in bands B2, B3, B4 and B5 have links to more bands than the source in band B1; therefore the link to S1 is severed (unless its link passes the looser threshold test). Then, the processing restarts from the beginning for the detection that revealed the linkage rejection originally.
After all bandmerging processing is finished, each group of merged detections is processed for position refinement. This proceeds by identifying which detections are eligible to contribute their positional information to the refinement; if none are eligible, then the detection with the largest positional uncertainty (i.e. determinant of the position error covariance matrix) defines the final position. If only one detection qualifies, then its position parameters are used for the merged source. Otherwise, Gaussian refinement is performed using all qualified detections. Currently eligibility determination is not implemented, and all merged detections contribute their information to the refinement.
1.8 Statistical Summary Computation
The final processing step consists of producing a statistical summary. This is only done for Spitzer data. All multi-band (merged) sources are examined, one at a time, and only those that are completely free from confusion are considered for inclusion in the statistical summary. If the optional input column signal-to-noise ratio has been included as part or the current processing run, the SNR values for all point sources in each merged source are examined. If at least one of the point sources included in a merged source has an SNR value above the threshold for that band, this merged source is included in the statistical data. For all band-pair combinations included in the current merged source, average delta-X and delta-Y, standard deviation in X and Y, as well as chi-square in X, Y, and X/Y, are calculated. This information is binned by signal-to-noise ratio and the totals (for all SNR bins) are also calculated. If, on the other hand, SNR values are not part of the current processing run, only the “Total” category is calculated. If any of the bands in a merged source has a null SNR value then this source is not included in the statistical data.
1.9 Iterative Processing
One of the main challenges with bandmerging is dealing with the positional offset between the different bands (especially when taken with different detectors). Bandmerge is designed to run as an iterative process, whereby the bandmerging module and the position refinement steps are run using a crude initial estimate of the offset between bands (given in the Bandpair uncertainty file; default bpru.tbl), and the output of the position refinement step is then used to modify the positional information before re-running the bandmerging module. This loop of bandmerging - positional offset modification - bandmerging is defined as a single iteration.
