Data Downloads from the Spitzer Heritage Archive

Contents of page/chapter:
+Overview
+Deciding what files to download
+The Background Monitor
+Help! I have a lot of zipfiles!
+Using the Download Scripts
+Help! These things are unzipping into separate directories!
+Acknowledging Spitzer and the SHA

Overview

After searching and investigating your results, you have presumably decided which data to download, and the next step is actually downloading it.

In the simplest case, just click the checkboxes on the far left of each row (from the AOR, Level 2, or Level 1 tabs) to pick specific data files to download, and then click "Prepare Download" to begin the packaging (and downloading) process. A pop-up window will appear in order to define exactly what kinds of data you would like to have packaged up. Clicking OK initializes this process.

To select all of the data you have displayed, click on the checkbox at the top of the column of checkboxes, and all of the rows are automatically clicked (even those on subsequent pages). Then click "Prepare Download" to initiate the packaging process.

From the pop-up window, you can enter an email at which you would like to be notified when the packaging process is complete. (Within the same SHA session, the SHA remembers what you have entered before, but when initiating a new SHA session, you will have to re-enter this information.)

The packaging process is spun off to the Background Monitor. Once the packaging is complete, to download the data, just click on the "Background Monitor" link, then click on any of the provided links reading "Download Now".

Note that you control where the data are saved on your disk through your browser; your browser may be configured to store all downloads in a particular location on your disk.

Deciding what files to download

All users should consult the corresponding Instrument Handbook (available at the IRSA Spitzer site) for details and for information to consider for your own project.

Most new users probably will want the most highly processed data versions available -- Enhanced Products first, then Level 2 (PBCD) products.

More experienced users, or users wanting more control over the analysis (e.g., wanting highly accurate photometry), will want to download the Level 1 (BCD) data.

Very experienced users may want the Level 0 (raw) data. Choose this option with caution! Note also that some raw data was not included in the generation of the final products. When selecting the 'raw data' download option, all of the frames associated with your AORs are downloaded, not just the ones, e.g., within a smaller search radius.

The Background Monitor

The Background Monitor appears in the upper right of your window to keep track of the data download. It actively changes to reflect what it is doing ("Preparing", etc.: ).

A pop-up window can be called up at any time by clicking the "Background Monitor" (or "Preparing", etc.). You can watch your data being packaged, and when the data are available for download, it will update that window, providing a link (or more than one link) for downloading the packaged data. It also provides a download script (optional) if you have more than one package to download.

Important note: don't log in while it is preparing a download! If you log in during a download, as far as the system is concerned, you have changed identities (from 'anonymous' to 'jones', say), and as a result it does not complete preparing the download.

The Background Monitor also keeps track of very large catalog queries, ones that take a long time to render in the SHA itself.

To stop any query mid-way through, click on the little red octagon ("stop sign").

If you forgot to put in your email at the beginning, or if the packaging is taking longer than you expected, you can click on "Add email" from the lower right of the Background Monitor popup and add your email during the packaging process.

Help! I have a lot of zipfiles!

When downloading large quantities of data (big programs, whole campaigns, etc.), the SHA will break up the downloads into "manageable" pieces, where "manageable" is defined as "not larger than common computers and software can handle." We understand your frustration if you computer is customized to handle much larger files than the average computer. If you don't want to click to download each piece, use the download script provided by the Background Monitor, available when the packaging is complete either in the Monitor itself or from the email you can arrange to have sent to you. The script can be configured to unzip the files too. See the next section for instructions on how to use it.

Using the Download Scripts

The Download Retrieval Script dialog gives you some options regarding which script you want to use. Generally speaking, the wget script is best for Linux and Unix users. The curl script is best for Mac users, because curl is part of the standard OS distribution; Mac users can also go retrieve and install wget and then use the wget scripts. For any of the scripts, you can also choose to include an option that unzips the zip files automatically. The files stay on disk here for at least 72 hours, so you have a window of time to download them.

Save the script to a plain text file, and invoke the script. You can copy and paste the script lines individually into your terminal window, or by typing "csh [yourtextfile]" at the prompt. The files will be automatically and sequentially downloaded to your disk, and if you've selected that option, unzipped as well.

For Windows users, download and save the text file of URLs . Then follow the following steps to install the wget script and then download your data:

  1. Go to the Windows wget web page
  2. Scroll to the Download section and retrieve the wget installation.
  3. Install wget and add the binary to your path.
  4. Download the text file of URLs
  5. At the command prompt: wget --content-disposition -i <file_of_urls_downloaded.txt>

Help! These things are unzipping into separate directories!

Depending on how, exactly, you unzip your files, your computer may put the contents of each zipfile into one directory, rather than, say, the contents of each AOR into one directory (which was the original design, and the conceptually most straightforward). If you are using a GUI-based method (e.g., click to uncompress), there should be a preferences option to allow you to uncompress subsequent zipfiles into the same root directory. If you use the download script above, the flags sent on the command line that unzip the files should put all files from the same AOR in the same directory.

Acknowledging Spitzer and the SHA

Now that you've got your data, you're going to write a great paper. Please add this to the acknowledgments at the end (where you can include or drop the "in part" as your case applies):
This work is based [in part] on observations made with the Spitzer Space Telescope, obtained from the NASA/ IPAC Infrared Science Archive, both of which are operated by the Jet Propulsion Laboratory, California Institute of Technology under a contract with the National Aeronautics and Space Administration.