Scanpi Results README File ========================== This is a general README file in the Scanpi workspace which explains the directory tree structure and file naming scheme of the entire workspace, where processing for that Scanpi request took place. The file "index_results.html" returns the user to the 1st set of Scanpi results web page. Note, there are on average around 1500 Scanpi output files per source, when all 4 bands are requested. Also note, if the input user parameter "Show Plots of Intermediate Steps" is selected, then an additional group of files is created. These files are indicated with a "(*)" below. Lastly, note that some files are generated on-the-fly (OTF) in the workspace when you click on certain links on the results page. These files are indicated with a "(**)" below. Please refer to the information below in order to navigate to the files you may find useful in the workspace. The 'directory tree' structure is described first, followed by the 'coaddition files', followed by 'scan tracks', then the 'individual scan' files and lastly the 'summary' information. Directory Tree Structure ------------------------ The location of this README file is in the "workspace" of files. This is the top-most directory location for output files for each user-requested source position. There are up-to four sub-directories, one for each band of processing requested. Almost all the files in the workspace have the prefixed "b" followed by a number, which refers to the "band" number: "b1" for band 1 (12 microns) "b2" for band 2 (25 microns) "b3" for band 3 (60 microns) "b4" for band 4 (100 microns) The individual scan files are in sub-directories, depending on the band number: b1/, b2/, b3/ and b4/. All coaddition scans and individual scans are prefixed with the band numbers "b1_", "b2_", "b3_", "b4_", depending on the band. Co-addition Scans ----------------- Co-addition (coadd) scans names start with the "b" followed by the band number, as described above. There are 4 types of coadd files: (1) Detector Weighted Mean ("_coadd_detmean_", aka scan 999), (2) Mean ("_coadd_mean_", aka scan 1001), (3) Median ("_coadd_median_", aka scan 1002) and (4) Noise Weighted Mean ("_coadd_detmean_", aka scan 1003). The 4 coadd's are created using the individual scan files listed in each "b#_coaddlist.tbl". For ease of describing all the files, band 1 (12 microns) will be used below. Band 1 coaddition scan names, prior to background subtraction or fitting: b1_coadd_detmean.tbl => In-scan distance vs detector-weighted mean coadd'ed flux b1_coadd_mean.tbl => In-scan distance vs mean coadd'ed flux b1_coadd_median.tbl => In-scan distance vs median coadd'ed flux b1_coadd_noisemean.tbl => In-scan distance vs detector-weighted mean coadd'ed flux For each of the 4 types of coadds, the following background fitting and source fitting tables are also created (using just "_coadd_mean_" as the example): (1) Background fitting files: b1_bgfit_coadd_mean.tbl => In-scan distance, background subtracted flux, fitted background flux values b1_bgfit_coadd_mean.xml(*) => XML file used for IRSA plotting of _bgfit_coadd_mean.tbl file as .jpg; using background non-subtracted datapoints. b1_bgfit_coadd_mean.jpg(*) => JPG plot of the in-scan distance vs flux of the _bgfit_coadd_mean.tbl; data points are in black and background fit is in red. b1_bgsub_coadd_mean.xml(*) => XML file used for IRSA plotting of _bgfit_coadd_mean.tbl file as .jpg; using background subtracted datapoints. b1_bgsub_coadd_mean.jpg(*) => JPG plot of the in-scan distance vs flux with background of the _bgfit_coadd_mean.tbl; data points (background subtracted) are in black and background fit is in red (that was used for subtraction). b1_bgfit_coadd_mean.tbl_data_nosigma(*) => Non-IPAC format table, in-scan distance and flux where the center source region has been removed and flux values more than 2.5 sigma have also been removed b1_bgfit_coadd_mean.tbl_data_nosigma.gif(*) => GIF plot of the in-scan distance vs flux of the _bgfit_coadd*.tbl_data_nosigma file with the background fit in green, and +/- 1 sigma levels in blue and magenta b1_bgfit_coadd_mean.tbl_data_nosource(*) => Non-IPAC format table, in-scan distance and flux where the center source region has been removed; no sigma clipping of data points has been taken into account. b1_bgfit_coadd_mean.tbl_data_nosource.gif(*) => GIF plot of the in-scan distance vs flux of the _bgfit_coadd*.tbl_data_nosigma file with the background fit in green, and +/- 1 sigma levels in blue and magenta b1_bgfit_coadd_mean.tbl_poly1(*) => Non-IPAC format table, in-scan distance vs 1st pass polynomial fit to the coadd flux. b1_bgfit_coadd_mean.tbl_poly1signeg(*) => Non-IPAC format table, in-scan distance vs fit of -1 sigma of 1st pass polynomial fit to the coadd flux. b1_bgfit_coadd_mean.tbl_poly1sigpos(*) => Non-IPAC format table, in-scan distance vs fit of +1 sigma of 1st pass polynomial fit to the coadd flux. b1_bgfit_coadd_mean.tbl_poly2(*) => Non-IPAC format table, in-scan distance vs 2nd pass polynomial fit to the coadd flux where 2.5 sigma outliers are clipped after the 1st polynomial fit. b1_bgfit_coadd_mean.tbl_poly2signeg(*) => Non-IPAC format table, in-scan distance vs fit of -1 sigma of 2nd pass polynomial fit to the coadd flux. b1_bgfit_coadd_mean.tbl_poly2sigpos(*) => Non-IPAC format table, in-scan distance vs fit of +1 sigma of 2nd pass polynomial fit to the coadd flux. (2) Source fitting files: b1_srcfit_coadd_mean.tbl => Final coadd'ed scan results table which contains the in-scan distance, flux, flux with background, fitted background, fitted template and fitted template with background. b1_srcfit_coadd_mean.xml => XML file used for IRSA plotting of _srcfit_coadd_mean.tbl file as .jpg b1_srcfit_coadd_mean.jpg.sh => Instructions file for on-th-fly plotting of _srcfit_coadd_mean.tbl file as .jpg b1_srcfit_coadd_mean.jpg(**) => JPG plot of the in-scan distance vs flux of the _srcfit_coadd_mean.tbl file; data values are in black, the background fit prior to subtraction is in red and the fitted template is in cyan. b1_srcfit2_coadd_mean.jpg(*) => Same as b1_srcfit_coadd_mean.jpg, except datapoints have had the background removed. b1_srcfit2_coadd_mean.xml(*) => XML file used for IRSA plotting of b1_srcfit2_coadd_mean.jpg b1_srcfitbg_coadd_mean.jpg(*) => Same as b1_srcfit_coadd_mean.jpg b1_srcfitbg_coadd_mean.xml(*) => XML file used for IRSA plotting of b1_srcfitbg_coadd_mean.jpg Scan Tracks ----------- Each scan used in the Scanpi processing run is plotted on an ISSA or DSS background image to show proximity to the source, path, and direction. Scan tracks are plotted per band, using the associated ISSA (bands 1, 2, 3 and 4) and DSS (DSS2 R band only) images as the backdrop; scan numbers (in red) indicate the start of the scan. Since scan tracks are generated on-the-fly, most of the files below are not in the workspace unless the user clicks on the ISSA or DSS 'Scan Tracks' link(s). ISSA background images are called "issa#.fits", where # is the band number. DSS "DSS2 R" band is used for all band background images; they are called "b#_scantracksDSSF1.fits" and "b#_scantracksDSSF2.fits", where # is the band number. The extension "F1.fits" image is the file copied from MAST with a multi-hdu format; the extension "F2.fits" image is a copy of the primary HDU of the "F1.fits" file. The list of scans per band is in the "b#_scans.tbl" file. The instructions on attaining the background image and plotting the scans on that image is in the "b#_scantracksDSS.jpg.sh" and "b1_scantracks.jpg.sh" (for ISSA) files. Here is a list of all the scan tracks associated files for band one: issa1.fits => ISSA background image (band 1) to place the scan tracks. b1_scantracksDSSF1.fits(**) => DSS2 R band background image in multi-hdu format. b1_scantracksDSSF2.fits(**) => DSS2 R band background image, primary hdu, to place the scan tracks. b1_scans.tbl => List of scans which need to be plotted for that band. b1_scantracks.jpg.sh => Instructions file for creating the ISSA jpg scan tracks. b1_region.dcx(**) => Description file for plotting the tracks for the ISSA jpg scan tracks. b1_scantracks.jpg(**) => ISSA scan tracks jpg; scan numbers are indicated in red, at the start of the scan; selected scans are in cyan and deselected scans are in orange. b1_scantracks.hdr(**) => ISSA scan tracks text file of just the ISSA fits header. b1_scantracksDSS.jpg.sh => Instructions file for creating the DSS jpg scan tracks. b1_regionDSS.dcx(**) => Description file for plotting the tracks for the DSS jpg scan tracks. b1_scantracksDSS.jpg(**) => DSS scan tracks jpg; scan numbers are indicated in red, at the start of the scan; selected scans are in cyan and deselected scans are in orange. b1_scantracksDSS.hdr(**) => DSS scan tracks text file of just the DSS fits header. Individual Scans ---------------- The files for individual scans are in subdirectories "b#/", where # is either 1, 2, 3, or 4, depending on the ISSA band (12, 25, 60 and 100 microns, respectively). When all 4 bands are requested by the user, 4 subdirectories are present with files at all the Scanpi processing stages. The following naming scheme is used for each band, for each scan. Scan names are created as follows: b#_sop_obs_det_suffix b#_###_###_##_suffix "b" followed by the band number, followed by the sop (Satellite Operating Plan) number, followed by the observation number, followed by the detector number, followed by a suffix which is uniq for every processing step. Here is a list of all the possible file names for one scan, for band 1, sop=321, obs=045 and detector=53: b1_321_045_53_scan.tbl => IRAS Level 1 data file of ra/dec and flux at each read position by that detector. b1_321_045_53_offset.tbl => Calculated in-scan distance from source of each position of the "_scan" file. b1_321_045_53_offset.xml(*) => XML file used for IRSA plotting of _offset.tbl file as .jpg b1_321_045_53_offset.jpg(*) => JPG plot of the in-scan distance vs flux of the _offset.tbl file b1_321_045_53_rereg.tbl => In-scan distance and flux from _offset table, but placed on 0.1 arcmin spacing; the process of reregistration is done using cubic spline fitting. b1_321_045_53_rereg.xml(*) => XML file used for IRSA plotting of _rereg.tbl file as .jpg b1_321_045_53_rereg.jpg(*) => JPG plot of the in-scan distance vs flux of the _rereg.tbl file b1_321_045_53_bgfit.tbl => In-scan distance, background subtracted flux, fitted background flux values b1_321_045_53_bgfit.xml(*) => XML file used for IRSA plotting of _bgfit.tbl file as .jpg b1_321_045_53_bgfit.jpg(*) => JPG plot of the in-scan distance vs flux_wbg (flux with background) of the _bgfit.tbl file; red is the background level (YFIT column), while black is used for the datapoints (flux_wbg) before background subtraction. b1_321_045_53_bgsub.xml(*) => XML file used for IRSA plotting of _bgsub.tbl file as .jpg b1_321_045_53_bgsub.jpg(*) => JPG plot of the in-scan distance vs flux of the _bgfit.tbl file; red is the background level (YFIT column), while black is used for the datapoints (flux) after background subtraction. b1_321_045_53_bgfit.tbl_data_nosource(*) => Non-IPAC format table, in-scan distance and flux where the center source region has been removed; no sigma clipping of data points has been taken into account. b1_321_045_53_bgfit.tbl_data_nosource.gif(*) => GIF plot of the in-scan distance vs flux of the _bgfit.tbl_data_nosource file with the background fit in green, and +/- 1 sigma levels in blue and magenta (no sigma clipping has been performed on the data) b1_321_045_53_bgfit.tbl_data_nosigma(*) => Non-IPAC format table, in-scan distance and flux where the center source region has been removed and flux values more than 2.5 sigma of the background fit have also been removed b1_321_045_53_bgfit.tbl_data_nosigma.gif(*) => GIF plot of the in-scan distance vs flux of the _bgfit.tbl_data_nosigma file with the background fit in green, and +/- 1 sigma levels in blue and magenta b1_321_045_53_bgfit.tbl_poly1(*) => Non-IPAC format table, in-scan distance vs 1st pass polynomial fit to the flux. b1_321_045_53_bgfit.tbl_poly1signeg(*) => Non-IPAC format table, in-scan distance vs fit of -1 sigma of 1st pass polynomial fit to the flux. b1_321_045_53_bgfit.tbl_poly1sigpos(*) => Non-IPAC format table, in-scan distance vs fit of +1 sigma of 1st pass polynomial fit to the flux. b1_321_045_53_bgfit.tbl_poly2(*) => Non-IPAC format table, in-scan distance vs 2nd pass polynomial fit to the flux where 2.5 sigma outliers are clipped after the 1st polynomial fit. b1_321_045_53_bgfit.tbl_poly2signeg(*) => Non-IPAC format table, in-scan distance vs fit of -1 sigma of 2nd pass polynomial fit to the flux. b1_321_045_53_bgfit.tbl_poly2sigpos(*) => Non-IPAC format table, in-scan distance vs fit of +1 sigma of 2nd pass polynomial fit to the flux. b1_321_045_53_srcfit.tbl => Final single scan results table which contains the in-scan distance, flux, flux with background, fitted background, fitted template to the source and fitted template to the source including the background level. b1_321_045_53_srcfit.xml(**) => XML file used for IRSA plotting of _srcfit.tbl file as .jpg b1_321_045_53_srcfit.jpg(**) => JPG plot of the in-scan distance vs flux of the _srcfit.tbl file; data values are in black and fitted template in red (background has been subtracted). b1_321_045_53_srcfitbg.xml(*) => XML file used for IRSA plotting of _srcfit.tbl file as srcfitbg.jpg b1_321_045_53_srcfitbg.jpg(*) => JPG plot of the in-scan distance vs flux_wbg (with background, in black) and fitted background (YFIT) in red, and fitted template with background (TMPFIT_wbg) in cyan. b1_321_045_53_srcfit2.xml(*) => XML file used for IRSA plotting of _srcfit.tbl file as srcfit2.jpg b1_321_045_53_srcfit2.jpg(*) => JPG plot of the in-scan distance vs flux (without background, in black) and fitted background (YFIT) in red, and fitted template without background (TMPFIT) in cyan. Note that each scan can have up to 30 associated Scanpi processing files. If a scan only contains a _scan.tbl and _offset.tbl file, this indicates that the closest cross-scan distance of this scan was larger than the minimum user requested cross-scan distance. Therefore, that scan was left out of the rest of the processing (background, source fitting and coaddition). Summary Information ------------------- A number of other files are in the workspace, such as log files, summary tables and html visualization pages of the results. Here is a summary of these files. wget_data.bat => Bulk download script, which contains a "wget" command which downloads all the files in the workspace recursively. iteration.txt => File contains one number, the current iteration number of the source being processed by scanpi (number of times this source was run/re-run). logfile.txt => Log file containing the background and source fitting parameters. summary.html => Scanpi results file which is reached from the 1st results page by selecting "View, Select, Rerun"; contains summary tables of all requested bands; user can rerun any band/scan(s) using this page. b#_summary.tbl => IPAC table, per band, presented in the "View, Select, Rerun" Scanpi results page, per band. b#_summary_sh.tbl => A subset of the b#_summary.tbl which is presented on the first Scanpi results page (this is a short/condensed version of the summary table). b#_summary.html => HTML version of b#_summary.tbl. b#_summary_sh.html => HTML version of b#_summary_sh.tbl dCross.tbl => IRAS Level 1 file which lists all the scans which crossed near the requested user position. The columns sop, obs and det are used to create the individual scan name files. index_results.html => Scanpi results file first displayed for an individual source run. stepbystep.html(*) => If the user selects "Show Plots of Intermediate Steps" on the Scanpi front page, they can see all the step-by-step processing and plots from the results page, using the "Plots of Intermediate Steps" link. This web page contains all the intermediate plots for all requested bands. b#_step.tbl(*) => IPAC table, per band, of all the intermediate plot links for the stepbystep.html summary. b#_step.html(*) => HTML version of b#_step.tbl b#_step_sm.tbl(*) => IPAC table, short version of b#_step.tbl, listing just the scan number and link to the table name. README => This file.