SPHEREx Data Explorer at IRSA: Downloading Data

Overview

To download data, in the simplest case, on the search results page, just click the checkboxes on the far left of each row to pick specific observations to download, and then click the "Generate Download Script" button to begin the process. A pop-up window will appear in order to define exactly what kinds of data you would like to download. Clicking "Download Script" in the pop-up initializes this process.

To select all of the data you have displayed, click on the checkbox at the top of the column of checkboxes, and all of the rows are automatically clicked (even those on subsequent pages if you have more than one page). Then click "Generate Download Script" to initiate the process.

The packaging process spins off into the Job Monitor, which keeps track of its progress and notifies you when the downloads are complete. You can choose to have an email sent to you to let you know when things are ready, even after the packaging process has started.

For the Spectrophotometry Tool, the button reads "Prepare Download" rather than "Generate Download Script", but the overall behavior is the same.

⚠ Tips and Troubleshooting

• You control where the file is saved on your disk through your browser; your browser may be configured to store all downloads in a particular location on your disk. Try looking in a "Downloads" directory, or for "recently modified files."
• You need to be logged in to use the IRSA Workspace .
• Mosaics are single images -- even though they are multi-plane, multi-HDU images, the tool still thinks of them as single images. They don't have a corresponding table and "Download Script" button, so just go up to the upper right of the image, click on the tools icon and then the diskette to save the image.

Options for Downloading Data

From the pop-up, you can choose what to download.

Title

The name by which this packaging job will be known to the Job Monitor as well as what the root filename is for the saved file - change it to whatever you want (time or a description of your search is usually the most helpful).

Products

What, exactly, would you like to download? Read the documentation linked here and decide what you need.

Flattened file structure

The observations can be bundled one per subdirectory (structured) or all in one directory (flattened).

File location

Where do you want the script to be saved? It can be saved in a file on your local disk or the IRSA Workspace .

⚠ Tips and Troubleshooting

• If you select cutouts to download, it will provide for you cutouts of the same size as you currently have displayed in the tool. If you want to have cutouts that are a different size, you need to change what is displayed in the tool before intiating the download. It will remind you of this:
  

Job Monitor

Here is a well-populated job monitor list:

At the top, it summarizes (in this case) that there are 14 jobs, two of which are currently active and executing, and two of which have failed. The email has been filled out, so that email notification can easily be toggled on (or off) with a single click. By default, it shows created, start, and end times in UT, but you can toggle it to match the same time as wherever your computer is currently; in the screenshot above, it's Pacific Standard Time (PST).

Below that, there is a table, like any other table in this tool (with all the associated sorting and filtering cabilities), that lists all of the jobs that have been sent to the Job Monitor in this session, along with basic information like what time (in Universal Time) they were created, started, and ended. One in this screenshot is generating a data download script (type = script), and it is still executing. There are other jobs that are various kinds of searches (type = SEARCH or TAP, the latter of which is Table Access Protocol ), and analysis jobs (the spectrophotometry tool is type = UWS -- UWS is Universal Worker Service ).

On the far right is a collection of icons. The different icons do different things:

	Usually this icon is animated -- the job is executing.
	Display the results of this job in the tool -- this icon matches that in the "Results" tab, which is where the results will appear after you click on this icon.
	Notification toggled on -- if an email is provided in the top of the Job Monitor, then email will be sent upon completion (see below).
	Notification toggled off -- this is the default state.
	Download the results of this job to your disk -- this could be a zip file, a data file (e.g., a FITS file), or a download script, depending on what data you are accessing.
	Show the choices for a download script, depending on what data you are accessing.
	Get more information about this job (see below).
	Stop this job.
	Discard this job from this list.

Jobs accumulate in the Job Monitor over a given session, which could include more than one browser window accessing the same archive, because it relies on cookies being set. Jobs that are older than 2 weeks will not appear. Some jobs (particularly TAP jobs) may only persist for up to 72 hours. Jobs that are launched via a different browser type (e.g., Chrome vs. Firefox) or on a different computer will not appear.

You are allocated two running spectrophotometry jobs at a time. If you submit more than two spectrophotometry jobs, the additional jobs will appear as "queued" (as seen in this screenshot) until one of your running jobs clears. The "start time" tells you when the job actually started, as opposed to the "created time", which is when you submitted the job.

⚠ Tips and Troubleshooting

• You control where the file is saved on your disk through your browser; your browser may be configured to store all downloads in a particular location on your disk. Try looking in a "Downloads" directory, or for "recently modified files."
• Jobs that are older than 2 weeks or launched on a different browser or computer will not appear in the job monitor. Some jobs (particularly TAP jobs) may only persist for up to 72 hours. If servers had to be rebooted at the IRSA end, jobs may be lost, though we are working to mitigate this.
• Logging in before submitting jobs seems to mitigate lost jobs.
• You are allocated two running spectrophotometry jobs at a time. If you submit more than two spectrophotometry jobs, the additional jobs will appear as "queued" until space is available.

Sending Email

In order to make this happen, you have to do two things.

1. Enter a valid email at the top of the Job Monitor page. (It should go without saying that if you don't enter a correct email, it can't do anything to fix that.)

2. Tell it that you want it to send you an email for the specific job in question. Click on the blue bell at any time before the job completes to toggle an email being sent to you upon completion.

   	This means don't send email
   	This means do send email

   The reason it defaults to "don't send email" is so that you avoid spamming yourself - if you make a lot of requests, you most likely don't want it to send you a lot of emails in rapid succession.

⚠ Tips and Troubleshooting

• By default, it will NOT send you email when it is complete, even if you have an email filled out. (This is to avoid accidentally spamming yourself!) Click on the blue bell at any time before the job completes to toggle an email being sent to you upon completion.

Scripts

To use these, save the script to a plain text file, and invoke the script. You may need to do something at the commmand line like "chmod +x script.sh" and then "./script.sh" to invoke it. You could also copy and paste the script lines individually into your terminal window.

Generally speaking, the wget script is best for Linux and Unix users. The curl script is best for Mac users, because curl is part of the standard OS distribution; Mac users can also go retrieve and install wget and then use the wget scripts.

Double-clicking on the downloaded zip files should uncompress them, and then you should be off and running. However, some Windows users have reported having difficulty unzipping files using the default application. We recommend using 7-zip .

Job Information

	The information you get when you click on the i-in-a-circle in the Job Monitor looks something like this, which is for a spectral image search. It specifies things about the query, like when it was submitted and how long it took, but also the job ID (useful for helpdesk tickets), and the parameters you used (which you can copy using the ellipsis, just like for table cells), and also what information you provided -- username if you logged in, email if you provided it, title if you provided it, and sometimes the URL that links directly to the job (also useful for helpdesk tickets; to copy it, click on the tiny clipboard icon).
	It could also look like this, where this is for a spectrophotometry job. It includes things about the specified parameters and how long the runtime was, among other things. Here, again, the UWS job id is very useful for helpdesk tickets -- to copy it, click on the tiny clipboard icon.
	This is for a mosaic job. It also includes things about the specified parameters and how long the runtime was, among other things. To copy the UWS job id for helpdesk tickets, click on the tiny clipboard icon.