Frequently Asked Questions (FAQs) about the Spitzer Heritage Archive (SHA)

Interface questions

What does SHA/AOR/IER/BCD/PBCD/Level 1/Level 2 mean?
Here are some common definitions. The archive of Spitzer data is the Spitzer Heritage Archive (SHA). An individual Spitzer observation sequence is an AOR, or Astronomical Observation Request. In certain cases (often calibration or sometimes science observations), you may also see an IER, or Instrument Engineering Request. Either one involves many individual frames. The individual data frames that emerge, calibrated, from the Spitzer pipeline are Level 1, or Basic Calibrated Data, or BCDs. The products that come from combining these individual data frames (such as mosaics of individual pointings) are Level 2, or post-BCD, or PBCD data. Enhanced products come from combining AORs or doing post processing (such as synthetic photometry from spectra or source extraction from images). These can be contributed by the community, or generated by the SSC itself.

Do you have any tutorial videos?
Yes. The IRSA YouTube channel has several tutorials that are relevant to the SHA -- see the Spitzer playlist as well as the set of "micro-tutorials" relevant for more than one archive. The videos are in HD; you may need to manually use the YouTube gear menu to force it to realize this. There is also English closed-captioning available.

What's the file format that I need to use for the search-by-file upload?
See discussion here.

Why am I getting search results that are shaded red, which I can't download?
The red-shaded data are still proprietary. If they are your data, then log in and you should be able to download them. (If you need to create a new account in order to log in, you may have to interact with the Spitzer Helpdesk to get your new account tied to your new data.) If they are not your data, you can download those data after the data become public, usually about 12 months after the data were taken (usually 12 months after the data were released to the observers). You can add the release date as one of the additional fields to display in the search results pane.

How do I download all of the data shown at once?
Click on the checkbox at the top of the column of checkboxes, at the top left of the search results pane. Then click on "prepare download."

How come I can't make a 3-color image from Spitzer data I am exploring interactively?
The visualization tools currently only allow you to create 3-color images that you load from disk or from the web. If you want to create 3-color images using Spitzer data, download the data to disk and then load them into the SHA from disk as individual color planes. Creating 3-color images within the SHA itself is on the wish list for future versions.

I know that there are a lot more files that should come as part of Level 1 (BCD) or Level 2 (PBCD) data. How come I can't see them in the Level 1 or 2 tabs?
Yes, there are a lot more files than what are shown, such as the errors! We decided for clarity that only certain files of primary interest to users would appear for individual examination in the SHA interface, but of course the download includes many more files -- you don't even have to tell it to include ancillary products. The IRAC, IRS, and MIPS Instrument Handbooks list (and define) all of the files that are included in the downloads.

How come it is only showing me mosaic files in the Level 2/PBCD tab? Where are the errors, shorter exposure frames for HDR data, etc?
Only the highest level products are shown in the Level 2/PBCD tab, and for that matter the Level 1/BCD tab. This is designed for you to inspect the data enough to see if you want to download it. If you package up the data for downloading, the most frequently used standard files (errors, coverage, masks, etc.) will come with the download; to ensure you have all the files, select "incuding ancillary" when packaging.

I did a search, left the SHA window up, and now that I've come back to this session, it's in a funny state and does not appear to be responding as it should. Why?
Sometimes, if there is a network hiccup (e.g., you pick up and move your laptop to a different wireless hub), the SHA can be left in a funny state. (In some extreme cases, you will need to not just restart the browser but clear your browser cache.) To avoid losing a search, if you've accomplished a complex search, log in to the SHA and save or tag the search. That way, you can recover it easily with a single click the next time you log in.

Is there a way to interact with the SHA via a computer program? (Is there an application program interface, or API?)
Yes; learn more about it here.

I asked it for just MIPS data, but the SEIP tab is returning IRAC data.. What gives?
Many of the 'more options' obtainable for a position search don't apply to enhanced products of any sort. For example, if you ask the SHA to give you just IRAC observations at a particular position, but also ask it to give you IRS Enhanced Products (which are of course completely independent of any IRAC observations), it will return just IRAC observations in the AOR/Level 1/Level 2 tabs, but the IRS Enhanced Products search is independent of the AOR search, and may still return viable results in that tab. Similarly, if you restrict your search to MIPS results but still ask it to give you IRS Enhanced Products and SEIP Super Mosaics, it will give you IRS and IRAC data, respectively, in those tabs, regardless of the MIPS filter you have imposed (that MIPS filter applies just to the AOR/Level 1/Level 2 tabs).

File questions

It broke up my request into, like, 30 zip files! Is there any way I can get it in larger pieces, or somehow streamline the data download?
When downloading large quantities of data (big programs, whole campaigns, etc.), the SHA will break up the downloads into "manageable" pieces, where "manageable" is defined as "not larger than common computers and software can handle." We understand your frustration if you computer is customized to handle much larger files than the average computer. If you don't want to click to download each piece, use the download script provided by the Background Monitor, available when the packaging is complete either in the Monitor itself or from the email you can arrange to have sent to you. The script can be configured to unzip the files too.

The Download Retrieval Script dialog gives you some options regarding which script you want to use. Generally speaking, the wget script is best for Linux and Unix users. The curl script is best for Mac users, because curl is part of the standard OS distribution; Mac users can also go retrieve and install wget and then use the wget scripts. For any of the scripts, you can also choose to include an option that unzips the zip files automatically. The files stay on disk here for at least 72 hours, so you have a window of time to download them.

Save the script to a plain text file, and invoke the script. You can copy and paste the script lines individually into your terminal window, or by typing "csh [yourtextfile]" at the prompt. The files will be automatically and sequentially downloaded to your disk, and if you've selected that option, unzipped as well.

For Windows users, download and save the text file of URLs . Then follow the following steps to install the wget script and then download your data:

  1. Go to the Windows wget web page
  2. Scroll to the Download section and retrieve the wget installation.
  3. Install wget and add the binary to your path.
  4. Download the text file of URLs
  5. At the command prompt: wget --content-disposition -i <file_of_urls_downloaded.txt>

How long do these zip files stick around on IRSA's disk after they are prepared?
72 hours, sometimes longer if the SHA is not under heavy use.

What do I do with the zip files I get when I download data?
To uncompress the files you have downloaded, type "unzip foo.zip". To uncompress multiple files at once, type "unzip '*.zip'" (the single quotes are important), or "unzip \*.zip" -- you just have to escape out the wildcard.

What is the file naming scheme for all these files? Why do I have so many files for a single observation? Which files do I need to download my data?
Each of the instruments (IRAC, IRS, and MIPS) has an Instrument Handbook, which lists and defines each of the files. They also describe instrument artifacts, and give guidance as to how to reduce the data.

Why does my IRAC observation appear to have no post-BCD (Level 2) data?
If the data was taken in IRAC's subarray mode, or for certain IERs (Instrument Engineering Requests), there will be no post-BCD data for the observation. Data in IRAC subarray mode are not processed through the online post-BCD pipeline. The post-BCD pipeline was not designed to work with the three-dimensional data cubes that are the subarray BCDs. When the software was developed it was deemed unlikely that subarray observations would be used for anything other than photometry of single bright objects and mosaics would not be a desired product. As a result, post-BCD products (e.g., mosaics and source lists) are not available from the pipeline. Please see the IRAC Instrument Handbook for more information on dealing with subarray data.

Where are my dmask files?
Pipeline products after S18.5 no longer provide dmasks as they contain misleading and incomplete information. They have been superseded by the information contained in the imask files. The imasks are more robust, include flagging of various artifacts that are not present in dmasks and make full use of the saturation correction made by the pipeline. Please see the IRAC Instrument Handbook (Appendix A) for more information.

Why does my IRAC observation appear to have incorrect header information?
If the data was taken in as IERs (Instrument Engineering Requests), the pipelines will not necessarily run to completion, which means that there may be incorrect values of some header keywords.

Why does my MIPS 24+70 photometry observation appear to have no serendipitous 24 um data?
If the 70 um data was taken in MIPS-70's fine scale mode, no serendipitous 24 um data were obtained during the 70 um observation. Please see the MIPS Instrument Handbook for more information on dealing with these data.

Getting more help

See getting more help below.