SPHEREx Data Explorer at IRSA: Upload

You can upload tables to SPHEREx Data Explorer in several different contexts. This chapter consolidates all of that information into one place, even though it appears in the application in several entirely different places.

Contents of page/chapter:
+Introduction
+File Location
+Catalogs
+Region Files
+Images
+Spectra
+MOC Files
+Data Link Files
+UWS Job Files

Introduction

If you click on "upload" in any of a number of contexts, you'll get something like this:

From here, you can upload new files ("Upload Tables") or work with one of the tables that are already loaded into the tool ("Loaded Tables"). The tables that are already loaded can be lists of images or lists of sources.

However, you can also get an upload tab from the side menu that brings up a more globally capable upload tab:

Some things you can do from either upload interface, and some things can only be done from the upload tab.

File Location

The file that you are uploading can come from your own disk, the web (type or paste in the URL), or from the IRSA Workspace

. (Note that you need to be logged in to use the Workspace.)

Catalogs

You may upload a file from disk (you can use the system browser to identify the file, or drag-and-drop them into this window), from the web via a URL, or from the IRSA Workspace

The file can be in any of a number of formats, which we now briefly describe.

IPAC table format (*.tbl)

IPAC table format

is plain text with a particular formatting. IRSA has a table reformatting and validation service

which may be helpful, or you can download just about any catalog you find through IRSA, and mimic that format.

If you want it recognized as a catalog, your table file MUST have RA and Dec values, and unless it is specified, it assumes J2000. (See also "tips and troubleshooting", below.)

If you are uploading a catalog specifically for overlaying on an image, you can add a "SYMBOL" parameter to change the shape (X, SQUARE, CROSS, EMP_CROSS, DIAMOND, DOT) of catalog marks, e.g.:

\SYMBOL = X

You can add a "DEFAULT_COLOR" parameter to assign a CSS color name or a HEX value to catalog marks, e.g., either of these two:

\DEFAULT_COLOR = lightcyan
\DEFAULT_COLOR = #00FF00

You can find the CSS color code or the CSS color HEX values

online.

Comma-separated values (CSV) format (*.csv)

CSV format is often accepted by spreadsheet programs, and most spreadsheet programs can output CSV. It is plain text, with values for each column separated by commas.

Tab-separated values (TSV) format (*.tsv)

TSV format is sometimes accepted (or generated) by spreadsheet programs, and sometimes in tabular data downloaded from journal articles from the 90s. It is plain text, with values for each column separated by tab characters.

VOTable (*.vot)

Virtual Observatory (VO) tables are a special case of XML tables. (All VO Tables are XML but not all XML are VO Tables.) It is a format developed by the International Virtual Observatory Alliance (more information here

). XML tables initially look to the human eye like it might be HTML, but they are easier to parse with code. There are packages in astropy that handle VO Tables.

Parquet (*.parquet)

Parquet file format

is a highly efficient, compressed, column-oriented format for tabular data that has been adopted by many recent wide area survey projects. It can enable faster searching for large tables. There are packages in astropy that handle parquet files.

FITS files (*.fits)

FITS tables can be loaded into this tool. Note that this tool is flexible enough to handle multiple header data units (HDUs), so that you can upload a FITS file that has both images and tables. You can specify which HDU you would like to load.

ASDF (*.asdf)

ASDF file format

is an extensible and human-readable tree-based data structure for text and binary data that has been adopted by a few projects, namely JWST and Roman. There are packages in astropy that handle these files.

⚠ Tips and Troubleshooting: If you are trying, e.g., to upload a target list into the search screen, and you have multiple HDUs in a FITS file, you can load a catalog plane; it will ignore any image planes you try to load in this context.

Nearly every file you load will result in a preview of the file you have uploaded. Here are some example previews of catalog uploads:

Uploading a CSV file:

Uploading an IPAC table file:

⚠ Tips and Troubleshooting

In order to search on your list of objects, it needs to have RA and Dec columns. If the tool doesn't seem to recognize your RA and Dec columns, check your formatting, or try headers of "ra" instead of "RA" and "dec" instead of "DEC" or "Dec". Make sure those columns are being interpreted as numbers. If you have columns like "_RA2000" and "_DE2000", or it thinks those columns are strings (which it might, if there are NaN or null entries), it's going to be confused.
Unless specified, the tool assumes any coordinates you give it are J2000.

Region Files

DS9

is a popular program for visualizing FITS files. It uses a file format for storing image overlays called region files. The Eucid Data Explorer can write and read DS9 region files. Usually, you would read regions files from the image toolbar, but you can also upload region files via this upload tab.

If you upload a region file here, the tool will just assume that you meant to upload it via the image toolbar and overlay it on your image(s). If you don't have an image loaded, it will warn you.

⚠ Tips and Troubleshooting

If you want to overlay sources from a catalog, it is generally better to actually create a catalog (as above) and upload/overlay that, rather than a region file; you have more flexibility with what you can do (how you can interact) with a catalog than you can with a region file.

Images

You can upload single- or multi-plane FITS or ASDF files into the tool, as well as multi-HDU FITS files. Again, it will give you a preview of what it thinks you are uploading. For multi-plane or multi-HDU files, you can select what portion(s) to upload.

Uploading a simple FITS file:

Uploading a FITS file with many image extensions, with just one selected:

Uploading a FITS file with mixed extensions, with just one image plane selected:

Uploading a multi-HDU FITS file containing a mixture of tables and images. (Note that in this case, all planes are selected and the lower left gives a choice for loading all the images into one window or one extension per window. Note also that it has the option of attempting to interpret tables as spectra.)

After you verify that the tool is reading your file correctly, and, if applicable, selected the HDU(s) you wish to load, click "Load" to load the file into the tool.

The images are then shown and interacted with in the same way as the other images described here.

Spectra

You can upload tables and tell the tool to attempt to load them in and treat them as spectra. Again, it will give you a preview of what it thinks you are uploading. It can handle all the same file formats as above, but you may need to help it realize that you are giving it a spectrum.

Uploading an IPAC tbl file that is a spectrum, asking the tool to interpret the table as a spectrum:

Uploading a FITS file that has a spectrum included, asking the tool to interpret the table as a spectrum:

The spectra are then shown and interacted with in the same way as the other spectra described here.

⚠ Tips and Troubleshooting

By default, spectrophotometry results are saved as an xml file. You can read this xml file back into the tool using the upload tab. It won't behave exactly the same as if it were displayed from the job monitor, but you can still view and interact with it. See the spectra chapter for more information.

MOC Files

Multi-order coverage map (MOC)

files tell you where data exist (or don't exist). You can upload these kinds of files into this tool, and you can choose to view them as a table or as an overlay on HiPS files.

You can also load a MOC file from the HiPS/MOC menu once you have a HiPS image loaded.

A preview when uploading a MOC file :

Data Link Files

A DataLink

is a protocol developed by the International Virtual Observatory Alliance to specify more sophisticated linking of metadata and services to the data itself. You might use this kind of file to describe linkages to light curves or a light curve service from a single-epoch catalog. These kinds of files can also be loaded into this tool.

UWS Job Files

A Universal Worker Service (UWS) Pattern

is a protocol developed by the International Virtual Observatory Alliance to manage asynchronous execution of jobs on a service. These kinds of files can also be loaded into this tool.