SOFIA: Downloading Data

SOFIA Science Data Archive: Downloading Data

Contents of page/chapter:
+Overview
+Options for Downloading Data
+Job Monitor
+Sending Email
+Scripts
+Job Information
+Acknowledging WISE and IRSA

Overview

To save individual things like single tables, images, or plots, look for a diskette icon (

) in the corresponding pane. Tables, images, and plots can all be saved individually.

To download data, in the simplest case, on the search results page, just click the checkboxes on the far left of each row to pick specific observations to download, and then click the "Prepare Download" button to begin the packaging (and downloading) process. A pop-up window will appear in order to define exactly what kinds of data you would like to have packaged up. Clicking "prepare download" in the pop-up initializes this process.

To select all of the data you have displayed, click on the checkbox at the top of the column of checkboxes, and all of the rows are automatically clicked (even those on subsequent pages if you have more than one page). Then click "Prepare Download" to initiate the packaging process.

The packaging process spins off into the Job Monitor, which keeps track of its progress and notifies you when the downloads are complete. You can choose to have an email sent to you to let you know when things are ready, even after the packaging process has started.

⚠ Tips and Troubleshooting

You control where the file is saved on your disk through your browser; your browser may be configured to store all downloads in a particular location on your disk. Try looking in a "Downloads" directory, or for "recently modified files."
You need to be logged in to use the IRSA Workspace .

Options for Downloading Data

From the pop-up, you can choose what to download:

(The options may be slightly different, depending on the tab from which you initiate the download.)

The first row is the name by which this packaging job will be known to the Job Monitor as well as what the root filename is for the saved file - change it to whatever you want (time or a description of your search is usually the most helpful).

The second row shown here may not always appear, but is an indicator of whether you want any PNGs (e.g., preview images) or region files (created from overlays on your image). Not every download will really have these files, but the option will appear when inititated from an instrument tab.

The next row controls whether the observations are bundled one per subdirectory or are all in one directory (flattened).

The last row controls whether the file is saved to your local disk or the IRSA Workspace .

Click "Prepare Download" in the pop-up window, and it will go do it!

If you request relatively little data, the download will happen quickly. If you request a lot of data, you are given a choice to place the job in the Job Monitor.

⚠ Tips and Troubleshooting

If you initiate the download from, say, an EXES tab, you won't get FIFI-LS data. Make sure you are on the tab you want before initiating the download.
If you download from the AOR tab, you will get all the levels of data associated with that AOR. If you want only, say, Level 4 data, select only those files from the instrument tab.
Note that you control where the data are saved on your disk through your browser; your browser may be configured to store all downloads in a particular location on your disk. Look for a "Downloads" folder or search for recently modified files.
You will download zip files. Depending on how, exactly, you unzip your files, your computer may put the contents of each zipfile into one directory, rather than, say, the contents of each observation into one directory. If you are using a GUI-based method (e.g., click to uncompress), there should be a preferences option to allow you to uncompress subsequent zipfiles into the same root directory. You can also set flags on the command line to put all files from the same observation in the same directory.
Double-clicking on the saved downloads should uncompress them, and then you should be off and running. However, some Windows users have reported having difficulty unzipping files downloaded from the SOFIA archive using the default application. We recommend using 7-zip .

Job Monitor

To explore the Job Monitor, click on the Job Monitor tab at the top of the screen:

Here is a well-populated job monitor list:

At the top, it summarizes (in this case) that there are 13 jobs, one of which is currently active, and none of which have failed. The email has been filled out, so that email notification can easily be toggled on (or off) with a single click.

Below that, there is a table, like any other table in this tool (with all the associated sorting and filtering cabilities), that lists all of the jobs that have been sent to the Job Monitor in this session, along with basic information like what time (in Universal Time) they were launched. All of them in this screenshot are data packaging (type = package), one of which is still executing. Other kinds of jobs that could be in the background are catalog searches, but usually for SOFIA fields of view, the catalog searches are so fast that you don't have time to put them in the job monitor.

On the far right is a collection of icons. The different icons do different things:

	Usually this icon is animated -- the job is executing.
	Display the results of this job in the tool -- usually this appears in reference to a catalog search, and the icon matches that in the "Results" tab, which is where the results will appear after you click on this icon.
	Notification toggled on -- if an email is provided in the top of the Job Monitor, then email will be sent upon completion (see below).
	Notification toggled off -- this is the default state.
	Download the results of this job to your disk -- this could be a zip file, a data file (e.g., a FITS file), or a download script, depending on what data you are accessing.
	Get more information about this job (see below).
	Stop this job.
	Discard this job from this list.

Jobs accumulate in the Job Monitor over a given session, which could include more than one browser window accessing the same archive, because it relies on cookies being set. Jobs that are older than 2 weeks will not appear.

⚠ Tips and Troubleshooting

You control where the file is saved on your disk through your browser; your browser may be configured to store all downloads in a particular location on your disk. Try looking in a "Downloads" directory, or for "recently modified files."

Sending Email

For large data packaging jobs, you may wish to have the system email you when it is done and ready for downloading.

In order to make this happen, you have to do two things.

Enter a valid email at the top of the Job Monitor page. (It should go without saying that if you don't enter a correct email, it can't do anything to fix that.)
Tell it that you want it to send you an email for the specific job in question. Click on the blue bell at any time before the job completes to toggle an email being sent to you upon completion.
This means don't send email

This means do send email

The reason it defaults to "don't send email" is so that you avoid spamming yourself - if you make a lot of requests, you most likely don't want it to send you a lot of emails in rapid succession.

If you ask it to email you for a download packaging request, you will be emailed links to obtain a curl or wget script, or a list of URLs that you can feed to your own code to get your data. See the scripts section for more details.

⚠ Tips and Troubleshooting

By default, it will NOT send you email when it is complete, even if you have an email filled out. (This is to avoid accidentally spamming yourself!) Click on the blue bell at any time before the job completes to toggle an email being sent to you upon completion.

Scripts

If you get an email from the Job Monitor, you will get a curl or wget script that you can execute on your computer to download the files you have requested, or you may choose to obtain just a list of URLs that you can feed to your own code to download the data. The scripts are designed to echo to the screen updates on what they are doing, so that you can track the download's progress.

To use these, save the script to a plain text file, and invoke the script. You may need to do something at the commmand line like "chmod +x script.sh" and then "./script.sh" to invoke it. You could also copy and paste the script lines individually into your terminal window.

Generally speaking, the wget script is best for Linux and Unix users. The curl script is best for Mac users, because curl is part of the standard OS distribution; Mac users can also go retrieve and install wget and then use the wget scripts.

Double-clicking on the downloaded zip files should uncompress them, and then you should be off and running. However, some Windows users have reported having difficulty unzipping files downloaded from the SOFIA archive using the default application. We recommend using 7-zip .

Job Information

The information you get when you click on the i-in-a-circle

in the Job Monitor looks something like this:

It specifies things about the query, like when it was submitted and how long it took, but also the job ID (useful for helpdesk tickets), and the parameters you used (which you can copy using the ellipsis, just like for table cells), and also what information you provided -- username if you logged in, email if you provided it, title if you provided it, and the URL that links directly to the job (also useful for helpdesk tickets; to copy it, click on the tiny clipboard icon).

Acknowledging SOFIA and IRSA

Please consult this page

to find the acknowledgments to use for SOFIA.

Individual data sets at IRSA each have their own DOI .

Note that you can use the \facilities call in AASTeX to acknowledge both IRSA and SOFIA as a facility you used in your journal article.

	This means don't send email
	This means do send email