Using Tables with IRSA Services
Version 1.2 5/23/13
Many IRSA services allow the option to upload a table (Table Upload or Multi-Object Search) rather than entering objects one-by-one into a web form. The input table must be a plain text ASCII file, describing the search targets or positions. Several formats are allowed, but not all services accept all formats. This page describes the allowed formats and includes sample tables.
Hint: All IRSA services accept tables in IPAC Table Format with right ascension and declination specified in J2000 decimal degrees and labeled as lowercase "ra" and "dec" in the 1st header line. However Catalog Searches ("Gator") and the WISE Image Archive currently have restrictions on parts of the table, and the table filename. And most services will not accept a table filename with a space in it (allowed by the Spitzer Heritage Archive and WISE Image Archive).
There is an IPAC Table Validator that can check the format of an uploaded table and convert from the formats below (except for the special SHA Space-Delimited format) to IPAC Table Format with J2000 decimal degrees. See the first section.
Go Directly To A Section:
- IPAC Table Validator
- Table Formats
- Coordinates and Object Strings
- Best Practices for Successful Table Upload
- Troubleshooting Tables
- Contacting IRSA's Help Desk
IPAC Table Validator
The IPAC Table Validator 1) checks whether an uploaded table is in IPAC Table Format; 2) checks whether it satisfies the additional restrictions currently needed by the Catalog Search; and 3) if requested, tries to reformat the table to IPAC Table Format, and then re-checks (1) and (2).
To run, simply choose a table file from your local disk and hit Upload. The messages should be self-explanatory. Before worrying about the warnings, let it do (or try) the reformat step to an IPAC table (it will put a "_ref" in the name), which you should then be able to use in all IRSA services.
By default, it employs IRSA's name resolution/coordinate transformation to try to assign an RA and Dec to each target when "ra" and "dec" are not present (if not already on, click "Apply name resolution/coordinate transformation"). See the section below on Coordinates and Object Strings.
It's not always necessary to run the Table Validator. The next section explains which table formats each service can accept. But often it's easiest to run it and have a table in IPAC Table Format.
Table Formats
IRSA services accept some combination of the following ASCII table formats:
- IPAC Table Format (Also known as IPAC ASCII Column-Aligned Format)
- Comma-Separated Values (CSV) or Comma-Delimited
- Tab-Delimited
- SHA Space-Delimited
- A Simple List of Objects (such as astronomical source names or coordinates)
1. IPAC Table Format (or IPAC ASCII Column-Aligned Format).
The format definition is here: IPAC Table Format.
Briefly, an IPAC Table has the following structure:
- Header information for each data column is marked by vertical bars ("|") at the column boundaries.
- Up to four header rows are allowed, defining: Column name, data type, units, and null value.
- Data should lie within the cells defined by the bars in the header.
- Comments and keywords may be defined before the header in lines beginning with a backward slash ("\").
-
IRSA Services that accept this format:
- Atlas
- Catalog Search ("Gator") -- requires "ra" and "dec" in decimal degrees (J2000), and compliance with some other restrictions.
- Cutouts
- Dust Extinction
- Finder Chart
- Finder Chart 2 Beta
- WISE Image Archive
- Spitzer Heritage Archive
- 2MASS Extended Sources
- 2MASS Images (Batch Image Service)
- Scanpi (for IRAS data)
- SWAS Spectrum Server
Example of a basic IPAC Table Format file:
| id | ra | dec |
| char | double | double |
M56 289.147941100 30.184500500
ic4710 277.158208330 66.982277780
ngc 4552 188.915863750 12.556341390
2. Comma-Separated Values (CSV) or Comma-Delimited Format (Download example.)
This is a simple format where column names and data cells are separated by commas. Users storing data in a spreadsheet in Microsoft Excel often save their table in a .csv file.
Requirements for CSV tables include:
- The first line must contain the names of each column.
- The columns in the header row are separated by commas (",").
- The data cells in the table are separated by commas (",").
- The number of cells containing data is equal to or less than the number of columns.
- Blank data rows are treated as blank data records, so omit them to reduce processing time.
IRSA Services that accept this format:
- Atlas
- Cutouts
- Dust Extinction
- Finder Chart
- Spitzer Heritage Archive
- 2MASS Images (Batch Image Service)
- Scanpi (for IRAS data)
- SWAS Spectrum Server
Example of Comma-Separated Values (CSV) Format (also known as Comma Delimited):
id,ra,dec M56,289.147941100,30.184500500 ic4710,277.158208330,-66.982277780 ngc 4552,188.915863750,12.556341390
3. Tab-Delimited Format (Download example.)
This format is used, for example, when exporting data from Microsoft Excel to a .txt file and selecting Tab-Delimited. In this format, each cell is separated by a tab character. Tabs should not occur within cells, only between them.
Requirements for tab-delimited tables are virtually identical to those for comma-separated and include:
- The first line must contain the names of each column.
- The columns in the header row are separated by tabs.
- The data provided in the table are separated by tabs.
- The number of cells containing data is equal to or less than the number of columns.
- Blank data rows are treated as blank data records, so omit them to reduce processing time.
IRSA Services that accept this format:
- Atlas
- Cutouts
- Dust Extinction
- Finder Chart
- Spitzer Heritage Archive
- 2MASS Images (Batch Image Service)
- Scanpi (for IRAS data)
- SWAS Spectrum Server
Example of Tab-Delimited Format:
id ra dec M56 289.147941100 30.184500500 ic4710 277.158208330 -66.982277780 ngc 4552 188.915863750 12.556341390
In the above example, the columns are aligned using tab characters between each data cell. Note the data-columns may not line up exactly, even if tabs are applied consistently (as in row #3). Resist the temptation to add additional tabs in order to make the data align correctly; this will only create extra and empty columns, and your table upload will fail. Spaces are allowed in character data.
4. SHA Space-Delimited Format (Download example.) NOTE: only used by the SHA = Spitzer Heritage Archive.
In this format each cell is separated by a space. Therefore, if there is a space in your object name (e.g., "NGC 1001" versus NGC1001) or position ("34 23 45.45" versus 34d23m45.45s), you need to put quotes around the target name or its position. Since this format is only used by the Spitzer Heritage Archive, there are several additional requirements:
- The first line must specify the coordinate system.
- The second line must specify the equinox.
- The third line must specify the name resolver.
- No header is required.
- Lines need not have the same number of columns.
- Comment lines may be included with # as the first character.
Example of SHA Space-Delimited Format:
COORD_SYSTEM: Equatorial # Equatorial, Galactic, or Ecliptic - default is Equatorial EQUINOX: J2000 # B1950, J2000, or blank for Galactic - default is J2000 NAME-RESOLVER: NED # NED or Simbad - default is Simbad #Name RA/LON DEC/LAT PM-RA PM-DEC EPOCH "NGC 001" 12h34m23.45s 34d23m56.2s 2.3 3.4 1950.3 NGC2222 23.56d 34.456d 2.3 3.4 1950.3 NGC3333 23.56h 34.456d 2.3 3.4 1950.3 NGC4444 "12 34 12.23" "34 23 45.45" m31 legacy "17 18 00" "59 30 00" m32 m33 simbad NGC6946 NGC5194 ngc2992
5. Simple List of Object Names / Coordinates with No Header (Download example.)
Some IRSA services will accept a simple list of objects with no header. In this case, there are no complex formatting rules to remember. Objects can be listed by coordinates, or by name -- as recognized by NED and SIMBAD. Resolving the names with NED or SIMBAD may take a few seconds per row.
Requirements for this format are minimal:
- Each line has only one item.
- The entire line is treated as an object name or coordinate string.
IRSA Services that accept this format:
- Atlas
- Cutouts
- Dust Extinction (requires an |objname| header row; see Example 2 in the service documentation)
- 2MASS Images (Batch Image Service)
- Scanpi (for IRAS data)
- SWAS Spectrum Server
Example:
M56 ic4710 hoix 3 23 45.6 -12 34 56.78 ngc 4552 M82 3h 22m 15.6s -13d 52m 11.17s ngc1097 331.2 -6.94 ga M 65
Coordinates and Object Strings
Usually coordinates are input as separate table columns. Most services can also handle "object strings", defined below.
1. Coordinates in Table Columns
All IRSA services accept right ascension and declination specified in J2000 decimal degrees and labeled in the table with column names lowercase "ra" and "dec". This is the format one should use if possible.
The IPAC Table Validator can convert the table types above (except for the SHA Space-Delimited format), and a range of coordinate column names, to an IPAC table with "ra" and "dec" in J2000 decimal degrees. Other services can do this internally. Coordinate column names can be the following, in lower- or upper-case:
Coordinate column names accepted by the Table Validator
| Longitude | Latitude | Result |
| ra | dec | Equatorial J2000 |
| ra2000 | dec2000 | Equatorial J2000 |
| ra2000 | de2000 | Equatorial J2000 |
| _raj2000 | _dej2000 | Equatorial J2000 |
| raj2000 | dej2000 | Equatorial J2000 |
| raj2000 | decj2000 | Equatorial J2000 |
| elon | elat | Ecliptic J2000 |
| elon2000 | elat2000 | Ecliptic J2000 |
| elon1950 | elat1950 | Ecliptic B1950 |
| glon | glat | Galactic |
| l | b | Galactic |
Coordinates submitted to the IPAC Table Validator service can be any of the following:
Coordinates accepted by the Table Validator
| Longitude | Latitude |
| 50.94000 | -37.2614 |
| 3h23m45.6s | -37d15m41s |
| 3 23 45.6 | -37 15 41 |
| 3h 23m 45.6s | -37d 15m 41s |
| 50d 56m 24s | -37d 15m 41s |
2. Object Strings
If no coordinate columns are found, the IPAC Table Validator service checks the table for possible column names that might contain an "object string":
Object String column names accepted by the Table Validator
| Column Name |
| object |
| source |
| objname |
| objstr |
| locstr |
| location |
| star |
| galaxy |
| name |
NOTE: a valid coordinate column overrides an object string.
It either parses the coordinates or looks up the object by name with NED and SIMBAD. For conversion, click on "Apply name resolution/coordinate transformation". Coordinates in object strings can be decimal degrees or sexagesimal, Equatorial, Galactic, or Ecliptic:
Example Object Strings accepted by the Table Validator
| Object Strings |
| 3h23m45.6s -37d15m41s eq |
| 50.94000 -37.2614 ga |
| 50d 56m 24s -37d 15m 41s ec |
| M31 |
NOTE: Acceptable object strings are the same as the "Simple List" format above. They are also the same as those input to the single-source box.
Best Practices for Successful Table Upload
- Read the table upload help for the applicable service. Some IRSA services, such as Catalog Search ("Gator"), have stricter rules regarding table format.
- Run your table through the IPAC Table Validator service to convert your data to IPAC Table Format, especially if you intend to upload to Gator.
- Columns should not contain "non-printable" ASCII characters. Non-printable format control characters are sometimes inserted by text editing programs, e.g. LINEFEED (^J).
- Care is needed when using tabs. A TAB (^I) is a format control character indicating "jump to the next stop". In aligned formats, like IPAC Table Format, tabs can make the data look aligned (in your text editor) when in fact the characters don't line up. In tab-delimited tables, a tab is the separator between columns. In this format, any extra tabs will inflate the column count. It is usually safest to remove tabs from formats other than tab-delimited.
- Don't mix delimiters. For example, don't use a mixture of tabs and commas.
- Omit spaces in table file names. Most services (including Table Validator) will not read a file that has spaces in its name. For example, a file called "Spitzer Point Sources.txt" should be re-named "SpitzerPointSources.txt" or "Spitzer_Point_Sources.txt."
- Do not provide comment text (or anything else) at the top of the table unless you are using IPAC Table Format.
- The number of cells containing data should not exceed the number of column names. However, the number of cells can be fewer than the columns of data (the line will be padded with trailing blank cells).
Troubleshooting Tables
Error Message: Error opening table file.
Reason: The table may be in a binary file format, such as .doc, .rtf or .xls. In order for a table to be read by IRSA's services, it must be in a plain text ASCII file.
Solution: Ensure table is a plain text ASCII file. If using Microsoft Excel, tables saved as .txt files may still contain format control characters. Try saving the table as a .csv file. Many IRSA services accept this format (see above). For the others, run it through the IPAC Table Validator to get an IPAC Format table. Make sure the .csv has only one header row, containing the column names.
Error Message: We cannot identify columns to use as coordinates or a column to use as object names/locations.
Reason: The column names do not sufficiently describe either an object, or a location on the sky using ra and dec.
Solution: Clean up the column names so they are recognizable by IRSA services. See Coordinates and Object Strings for more information.
Example:
| jcg-ra | jcg-dec | size | 150.3814 2.3606 60.0 150.2794 2.1560 60.0 149.8873 2.0789 60.0 150.2323 1.9599 60.0 150.5407 2.5196 60.0 149.9343 2.4426 60.0
In the above example, there is extraneous text with ra and dec, rendering the column name unrecognizable.
Error Message: Failed to parse the table.
Reason: Often this is because the table contains non-printable ASCII characters.
A non-printable ASCII character in a table, such as ^[ (escape) or ^\ (file separator), can create problems during the upload process.
Solution: Remove the characters from the file. Viewing control characters is done differently in different operating systems:
- On Max OSX, Linux and Sun OS, use the Unix command cat to view a file's non-printable characters:
cat -tev filename
Here are two examples:
Example 1: cat -tev ipac_good_table.tbl
| lon | lat|$ | double| double|$ 237.140482 -28.797019 $ 237.3 -28.0 $
Example 2: cat -tev tab_delim_table.txt
object^Ira^Idec$ M56^I289.147941100^I30.184500500 $ ic4710^I277.158208330^I-66.982277780 $ hoix^I149.383333330^I69.045833330 $
This will bring your file to the screen with all hidden characters, such as tabs (^I), and end of lines ( $ ) indicated. - In Microsoft Word on Windows machines, you can view all tabs in a
document by toggling on formatting marks (click the toolbar button that
looks like a paragraph symbol, similar to the pi symbol). Remove the
arrows embedded within the table, and then save the file as "text
only." Here is a screenshot example:
Error: Search results return objects at coordinates different than the intended input coordinates.
Reason: One possible scenario is that the table is using IPAC ASCII Column-Aligned format, but coordinate columns with spaces are misaligned.
Solution: The table data should stay contained within the boundaries set by the vertical lines in the column header rows.
In the following example, a table with misaligned data is reformatted by the IPAC Table Validator, leading to an error.
Misaligned data columns:
| ra | dec | name| 12 24 36 45 29 27 m1 6 45 22 5 12 54 m2
Reformatted data columns:
| ra_u | dec_u | name | ra | dec | |char |char |char |double |double | 12 24 36 45 29 27 m1 186.1500000 +0.7580556 6 45 22 5 12 54 m2 101.3416667 +0.0866667
The first set of coordinates is interpreted as 12h24m00s, 00d45m29s instead of the intended 12h24m36s, 45d29m27s.
Contacting IRSA's Help Desk
If you having a specific issue with tables that is not addressed in this document, contact the IRSA Help Desk by submitting a User Support Ticket.