Spitzer Documentation & Tools
Spitzer Data Analysis Cookbook

 

 

Recipe 1.   Downloading Data from the Spitzer Heritage Archive

This recipe describes how to retrieve data from the Spitzer archive using the Spitzer Heritage Archive (SHA) web-based interface.

The SHA is housed at IRSA, the Infrared Science Archive. The URL is

http://sha.ipac.caltech.edu/applications/Spitzer/SHA/

1.1  Quick Hints, Tips, and Term Definitions:

  • An individual Spitzer observation sequence is an AOR, or Astronomical Observation Request. In certain cases (often calibration or sometimes science observations), you may also see an IER, or Instrument Engineering Request. Either one involves many individual frames.  Both have a large integer number called an AORKEY that uniquely identifies that observation within the mission.

 

  • The individual data frames that emerge, calibrated, from the Spitzer pipeline are Level 1, or Basic Calibrated Data, or BCDs.

 

  • The products that come from combining these individual data frames (such as mosaics of individual pointings) are Level 2, or post-BCD, or PBCD data.

 

  • Spitzer observations can cover large areas or, by design, multiple targets. If you are interested in just portions of the larger observation, you can choose to have just individual data products returned -- e.g., just the observations that went into the portion of the sky for which you searched -- or you can return the products for the whole AOR.

 

  • The SHA website displays subwindows, called panes.  Items within each pane often appear in tabs – click on different tabs to display different pane contents.

 

  • Also available are Enhanced Products that come from combining AORs or doing post processing (such as synthetic photometry from spectra or source extraction from images). These data can be a very powerful way to get started on using Spitzer data in your science pretty much straightaway. Enhanced products generated by the Spitzer Science Center (SSC) are returned in a tab separately from the substantial contributed enhanced products that were delivered by the community to the SSC and IRSA (and continue to be delivered to IRSA); these contributed enhanced products can include mosaics, photometry, spectra, and data from telescopes other than Spitzer.

1.2  Searching:

1.      Go to the SHA website (http://sha.ipac.caltech.edu/applications/Spitzer/SHA).  The first page is the search interface. It looks like this:

 

2.      Query the Spitzer Heritage Archive for the desired data set. This can be accomplished in a number of ways. By default, it comes up with a position search, as can be seen above.  On the left-hand side, several additional search options appear, including a link to the position search – you can search by abstract text, moving object, observation date, AORKEY (the integer number corresponding uniquely to each AOR), program, observer, or campaign.

 

Searching By Position: You can enter the name of the object and have the machine try either NED or SIMBAD first to resolve the coordinates, or you can enter the coordinates directly.

 

For this example, use:

RA 03 43 48.38 (J2000)

Dec +25 00 15.23 (J2000)

Which you can type in directly as : 03 43 48.38 +25 00 15.23

 

The SHA echoes back to you what it thinks you have told it, right below the box in which you type the name or coordinates. If you enter a name, by default it tries”NED first then SIMBAD” – if it incorrectly resolves it, select “SIMBAD first then NED” and it should re-resolve the coordinates in real time. If that is still wrong, enter the coordinates directly.

 

The search radius also can be changed, or you can leave it as the default.  For purposes of this example, change the "Radius (arcsec)" field to 5 arcmin by selecting “Arc Minutes” from the pulldown and entering 5 in the box. 

 

The next lines in the search window ask which kinds of data you want displayed: Display search results in tabs for: Observation Request (AOR), Level 2 (PBCD), Level 1 (BCD), and for the position search, also both SSC Enhanced Products (IRS), SEIP Super Mosaics, SEIP Source List and Contributed Enhanced Products.  Several of the other searches will allow you to select “SSC Enhanced Products” but not “Contributed Enhanced Products.”  By default, all searches return a list of AORs under the "AOR" tab, and a list of Level 2 products under the "Level 2 (PBCD)" tab (see figure below). You can also choose to access the original BCD frames by selecting the "Level 1 (BCD)" option here, and then the corresponding tab will be returned by your search.  Similarly, if you choose to search for enhanced products, those tabs may be returned; any given search may not have any enhanced products to return –  while there are IRS enhanced products scattered all over the sky, and while we have had the community return back to us products from observations all over the sky, there is much of the sky not covered by these observations. For this example, ensure that at the very least, the AOR and Level 2 (PBCD) options are checked, but you may also check both of the Enhanced Product boxes (SSC Enhanced Products (IRS), and Contributed Enhanced Products).

 

Finally, there is a “More options” line.  If you click on this, additional options will appear. You can choose to apply the spatial constraints to the AORs or individual data products – what this means is the following.  AORs can cover large areas or, by design, multiple targets. You can choose to return all AORs touching your search area, or just the individual data products enclosed within your search area – e.g., just the observations that went into the portion of the sky for which you searched.  For this example, check “AORs”. The next portion of the options allows you to search by instrument or wavelength, and, at the bottom, individual frame time and time of observation.  You may be interested in just the MIPS data, in which case you should unclick the “All IRAC” and “All IRS” boxes, and leave the “All MIPS” box checked.  For this example, leave it all as the defaults.

 

See the figure below for what your screen should look like at this point.

 

 

Then click on "Search" at the bottom of the browser window. The SHA will return 26 AORs for your consideration. You can now proceed to Step 3 below.

 

By AOR ID: Selecting “search by AORKEY” from the left array of options pulls up a very short window asking for the AORKEY. For this example, type in the ID number 5316352.  As for the other search options, you have choices for which tabs are returned by the search. Note that ‘Contributed Enhanced Products’ is not an option in this search because the contributed enhanced products are not necessarily tied to AORKEYs. For this example, ensure that at least the AOR and Level 2 options are checked; you can also select SSC Enhanced Products (IRS).  Click search.

 

 

Only one AOR will appear in the SHA search results pane. You can now proceed to Step 4.

 

 

By Program: Selecting “search by program” from the left array of options pulls up a shorter form, but with common elements to the position search. For this example, enter program "148"; you could also enter the program’s name.  As above, you have choices for which tabs are returned by the search. Note that ‘Contributed Enhanced Products’ is not an option in this search because the contributed enhanced products are not necessarily tied to programs. For this example, ensure that at least the AOR and Level 2 options are checked; you may also check SSC Enhanced Products (IRS).

 

As above, if you click on “More options”, you can choose to only display a subset of the data. Leave all of them on for purposes of this example.

 

Then click on "Search" at the bottom right-hand of the window.

 

 

 

The AORs associated with this program will appear in the SHA search results pane. You can now proceed to Step 3.

 

 

By Campaign/Observer: Selecting “search by Observer” from the left array of options pulls up a similarly short window. If you type in "Meyer", it will give you a list of options to auto-complete your selection.  Pick Michael Meyer’s name. As before, you have choices for which tabs are returned by the search. Note that ‘Contributed Enhanced Products’ is not an option in this search because the contributed enhanced products are not necessarily tied to an observer. For this example, ensure that at least the AOR and Level 2 options are checked; you may also check SSC Enhanced Products (IRS).  As above, if you click on “More options”, you can choose to only display MIPS data. For this example, leave them all checked.  Click on “Search” in the lower right.

 

 

Many AORs will appear in the SHA search results pane.  You can now proceed to Step 3.

 

3.      Find the AOR you want in the list of returned AORs under the AOR tab.  For this example, the one you want is observing a target called “HII_174” and is located at position (decimal RA, Dec) 55.951375 25.004417, or 03h43m48.33s +25d00m15.90s. 

 

Search results from the position search above look like this:

 

 

Things to note about this search result:

 

  • The type of search you are using is summarized at the top – in this case, a position search. To perform a different search, click on the “Spitzer searches” blue tab at the top left.

 

  • Your search results appear in two panes – search results and details. On the left are the search results in one pane, with an AOR tab and a Level 2 (PBCD) tab, and in this case “IRS Enhanced” and “Contributed Products” tabs; on the right is the “Details” pane, with tabs: Details, AOR DoC, and AOR Footprint. The contents of the Details pane changes corresponding to what you have selected (e.g., which tab and which green highlighted line) in the search results pane; more on this below.

 

  • If there are observations found that are still proprietary, e.g., not yet public, they will be highlighted by a red background color in the search listing. You cannot examine or download those data, unless you are (a) logged in, and (b) authorized by the PI to have access to those data; contact the Helpdesk if you are in this situation and still do not have access to those data.  Data become public at most a year after they were obtained, so in the worst case, you just have to wait.

 

  • These search results were found via the “search by position” example above, so if you did a different search, you may have more or fewer tabs and more or fewer or different AORs listed; more on this next.

 

Recall that the object we are looking for is a target called “HII_174” and is located at 3h43m48.33s, +25d00m15.9s.  Depending on exactly how you got to the results pane, you may need to find this object among many AORs. In order to do this, you have several options.  Note first how many search results you have to sift through; at the top of the search results pane, it tells you how many AORs are returned.  The default number of AORs displayed per page is 50, and you can adjust this as your needs dictate.  It is not generally recommended to have 1000s of AORs shown per page, because it will take too long for your browser to render the table. 

 

The top of the search results pane looks like this in the search results above:

 

In this case, there are only 26 AORs shown (“1-26 of 26”).  If you searched by PI name without restriction on the instrument, you could have more than 1000 AORs in the list “(1-50 of 1006)”.

 

To find your desired AOR among many, you can do any of several things, including:

 

a) Click on a column heading to sort by that column heading, then look for your AOR in the list. For example, in this case, we know our desired target is called “HII_174”, so we could sort by Target name and then find the page on which this target appears.

 

b) Impose a filter. Click on the “Filter Panel” (the funnel-looking symbol) at the top right of the results pane. You may need to grab the slider between panes and slide it over to see this.  The slider is the divider  with blue left-right arrows between the panes, annotated with a black arrow in this figure:

 

 

Click on the “Add filters” box at the top right of the results pane. Spaces will appear below column headers and you can enter an operator (if needed) and a value to compare. For this example,  enter in the AORKEY field “5316352” and type “enter” or “return”. (You could impose more than one filter in this process.) For this case, only one AOR should remain in the list after you impose the filter.  Filters are a powerful way of weeding down the list, and there are more examples of using them in the SHA manual.

 

4.      Get ready to download the data and do it.  Click on the checkbox on the far left of the row associated with this AOR (AORKEY 5316352, target “HII_174”).  Click on the “Prepare Download” button above this column in the search results window.  A popup appears asking what kind of data you want.  For this example, select the “Level 2, including ancillary” and “Level 1, including ancillary” products.  No need to download the calibration data or raw data. 

 

 

The total compressed data set takes up ~62 MB and should package very quickly.  On some browsers, it may be so small that it downloads all by itself.  The “Background monitor” will appear as a link on the top right of the SHA window, and you can watch it package if you click on it and look quickly. 

 

 

When it has completed, if the Background Monitor is not open, it will tell you it is ready:

When you open the Background Monitor, a “Download Now” link appears; click on it to download it. 

 

 

 

Your browser may put this file automatically somewhere on your disk, or it may ask you where you want it saved. This is a local, configurable option, controlled in your browser preferences.  If you can not find the download, look for recently modified files on your disk or look for a folder called something to the effect of  “Downloads.”

 

If you have a very big set of data to package up, and you did not enter an email when you did the packaging, you can ask during the packaging process for an email to be sent to you when it is done. Click on “Add Email…” in the lower right of this pop-up window in order to add an email; click on “Change Email” if you notice midstream that you have a typo or would like to change the address.

 

5.      Unzip the files that you have saved to your disk if your browser did not do it for you.  Your filename will vary, depending on exactly what search you did to get here.  You can unzip by double-clicking on the file, or type this in a terminal window:

unzip req-ID-5315584-1-selected_AORs.zip

 

6.      Figure out what all of these files are. Check the Spitzer web site for detailed documentation on data filename conventions and of course the instrument handbooks.

 

The data we have just downloaded may be easier to understand if we investigate the details of the observation itself.

 

In the main SHA search results window, select this observation.  (It may very well be the only one left in you search results window at this point.)  On the far right of the SHA window, there is a “Details” pane which contains many details about the observation.  The first tab, “Details”, gives you the original abstract and observer, as well as the instrument parameters that went into this specific observation, such as frame time.  The second tab, “AOR DoC”, is the AOR depth of coverage, e.g., how many individual frames were obtained per patch of sky for this observation. (Note it is number of frames, not total number of seconds.)  The third tab is “AOR footprint”.  Select that.

 

By default, the SHA attempts to pick an “appropriate” background image. For this observation, it selects a IRAS or DSS background image, and overlays red polygons on top of it. The red polygons outline the borders of the complete AOR.  The annotation at the top right of the image tells you the AORKEY that is overlaid, the nature of the background image (DSS POSS2 in this case), and the zoom factor (0.212x in this case).

 

 

If you let your mouse hover over the image, you will get a popup window with information about the image pixel underlying your mouse.  You will also see one icon appear on the top right – the one with arrows will “pop out” the image such that it takes up your whole browser window; there are also several tool icons along the top that allow you to manipulate the image (e.g., change the color table).  See the online SHA help for more information on each of the options, or let your mouse hover over an icon to get a “tool tip.”  Depending on what other visualizations you have been doing, the “pop out” of your image may show you your current as well as previous images. Select from the multi-view icons at the top of the popped out window to show more or fewer images; click on the “Close” arrow at the upper left to return to the search results view.

 

7.      Enhanced Products. If you selected either of the Enhanced Products options, you should have additional tabs to examine. By design for this example, this obervation was part of FEPS; FEPS, as a Spitzer Legacy project, was a large, cohesive observing program and also delivered observations back to the SSC. Also by design, this region also contains IRS Enhanced Products. If you selected both of those checkboxes for a position search, you should have those tabs as options:

 

Not all regions on the sky will return IRS Enhanced or Contributed Products. If your search does not return any data, no data will be shown in those tabs. For this example, though, you should find this for IRS Enhanced products:

For the IRS Enhanced Products, there is more information about those products in the IRS Instrument Handbook. You should find at least this for the Contributed Products:

 

 

 

 

So, FEPS delivered 12 images, and 2 spectra. They also delivered some photometry in two catalogs. To learn more about the deliveries (e.g., what exactly is in the delivery, quality flags, etc.), click on "more" (link on the right).  The "Data Delivery Document" is the main documentation accompanying the delivery.  To see what was delivered, click on the team name (link on the left).

 

In both cases, the “Details” pane on the right allows you to examine the data in more detail.

 

For the IRS Enhanced products, you can interactively explore the spectrum.

 

 

If you click on the two black arrows in the upper right of the spectrum viewer, you can get an expanded view:

 

A pop-up shows the values of the spectrum under your mouse.

 

For the Contributed Enhanced Products search results, this is a summary of all of the results it found. To see more, first click on a program’s name. It will spawn a tab that lists all of the contents, images in this case, that it found consistent with your search results:

 

If you select the “Data Coverage” out of the Details pane on the right, it will show an outline of all of the data products shown in the results pane, with the row you have selected (green highlights on left) also highlighted in the data coverage pane (blue highlights on right). 

 

For both of these enhanced products, you can visualize the data and select data for download in much the same fashion as for the rest of the SHA.  The table columns, especially for catalogs, can be cryptic, and are as provided by the teams. Consult the Data Delivery Document (or the IRS Instrument Handbook) for more information.