Catalog Search Tool: Tables

Contents of page/chapter:
+Interacting with Catalogs
+Hierarchical Catalog Display
+Table Header
+Table Columns
+Adding Columns
+Table Filters
+Row Details
+Table Cells
+Saving Tables
+Table Navigation

Interacting with Catalogs

Details of how to interact with the table is later in this chapter. Plots are covered in a different chapter, as are basic information about images and the visualization tools.

When you have more than one catalog loaded into the tool, the header of each catalog has the name of the catalog and a color swatch:

This color swatch corresponds to the symbol color that is used in the image overlays. You can change the color by clicking on the color swatch in the header, or by navigating to the layers in the image pane. See the color picker section of the visualization chapter for more information.

⚠ Tips and Troubleshooting

• Large catalogs will be displayed hierarchically! See next section.
• If you save the overlays from an image as a regions file, you may not get your complete catalog, especially if it is a large catalog (see next section!). However, you can save the full contents of a single catalog as a regions file using the "save" (diskette) icon in the table toolbar, instead of the image toobar.
• The "color swatches" may not appear immediately. To make loading faster, sometimes the colors don't load until they are actually needed.

Hierarchical Catalog Display

For catalogs above that threshold, the tool will bin up the catalogs based on HEALPix pixels (see HiPS section here for more links). In summary, the sky is broken up into sections, and the tool will show symbols with a number indicating the number of sources in that region. Then, when you zoom in, it will dynamically adapt to show you smaller and smaller cells until it shows you all the individual sources.

From the layers icon ( see visualization chapter), you can bring up many display options. Below are examples of what is displayed, the options seen in the layers, and additional options. The same catalog and zoom level and minimum group size are used for each view. The "Min Group" option here is 50, so if there are cells with fewer than 50 sources, then the individual sources are shown, and if there are more than 50 sources, then the cell is shown with a number inside corresponding to the number of sources from the catalog. (See below for additional information.)

	In this view, the 'cells' used are the cells explicitly associated with the HEALPix grid, so the size of the cells is very clear. In the top row here, three of the diamond-shaped cells across the top have fewer than 50 sources (so they do not have cell boundaries and the individual sources are shown), then the next row of diamond-shaped cells have 87, 86, and 57 sources respectively.
	In this view, the 'cells' are shown by circles enclosed within boxes. The locations and cell sizes are the same as in the prior screenshot, but the boundaries between tiles may be less obvious to new users.
	In this view, the 'cells' are shown by ellipses shown with dashed lines. The locations and cell sizes are the same as in the prior screenshot, but the boundaries between tiles may be less obvious to new users. It may be more obvious, though, that these are representations of groups of points.
	Finally, in this view, the 'cells' are again shown as the HEALPix tiles, but in this case the color of the cells corresponds to the number of sources in the cell. You can choose "Linear", "Linear Compressed", or "Log Stretch" to assign the colors, and you can change the color range by changing the color using the color picker in the layers pop-up, from which you can also change the transparency. This approach makes it more visually clear how many sources are in each cell, but makes it harder to see the background image. Even though you can change the transparency of this overlay to reveal more of the background, it still can make seeing the image challenging in some cases.

⚠ Tips and Troubleshooting

• For all of these renditions, when you zoom in close enough, it will dynamically adapt and show you individual sources when you zoom in. (That is, it no longer decimates the overlaid catalog, which is what it used to do.)
• For all of these renditions, if you click on a cell, it will display all of the sources in the cell. You can click on many cells in a row and it will continue to display all the sources it can until it reaches the point at which it thinks performance will suffer, at which point it will turn some of the points back into cells.
• If you want to have more of your catalog shown as individual sources, pick a smaller "min group" number.
• If you have more than one catalog loaded, the numbers within the cells (and in some cases the cell indicators themselves) will be offset slightly so that you can see them.
• If you have a catalog that includes sources from all over the sky, it very well may just give you box groupings, and may not allow you to change that view until you zoom in.
• If you have cells where only 1/4 of a cell is populated, it automatically renders a smaller cell, so if you have a sparsely populated but still large catalog, the size of the display will always be "small" size cells.
• If you are looking at many footprints from, say, a complex, and long ObsCore search, if you have more than 30,000 footprints, it may not be able to render all of the outlines of all of those images. It may render the centers of all of those images as if it were a catalog, in which case you will encounter these kinds of hierarchical catalog display options.

Table Header

The table is shown exactly as it has been provided to the tool, with all columns as defined by the creator of the table. To understand what each column is, please see the documentation associated with that catalog or table.

It can be that the table has a tab with a name on it. The tab (and table) name itself likely indicates its origin. It may be that you can remove the tab; if so, you can click on the "X" on the tab to remove it.

Immediately below the tab name, there can be several symbols:

which we now describe, going from left to right along the top of the catalog tab.

Table navigation

The first thing to notice is that (typically) only the first 100 rows of the retrieved catalog are displayed in the table. In the example here, there are 319 sources that were retrieved as a result of the search, grouped into 4 'pages.' The left/right black arrows plus the page number allow you to navigate among these 'pages' of 100 sources each. Note that the entire set of results (not just the 100 rows you are currently viewing) can be sorted by clicking on any column's name.

Filter

Filters are complex and powerful enough that they are covered in a separate section below.

Table as text

Clicking on this changes the table display into a fixed-width text display. The icon then changes to click this again to return to the default table view.

Save table

This is how you can save the table. It has a separate section below.

Add a column

This icon adds a new column to the table. This has a separate section below.

Info link

You may or may not see this icon. This is an "information" button and, if it exists, it may provide additional information about the table. It could have information about the job that was used to retrieve it:

where the direct link to the job is given there (and can be copied by clicking on the clipboard, ready to be pasted into a helpdesk query, for example), with a job id as shown. It could also just have information about the table metadata:

where the information about this table includes information about the query that produced it.

Row details

You may or may not see this icon. This is how you get more information about the currently-selected row. It has a separate section below.

Table options

Clicking on this icon brings up options for the table, e.g., how many rows are displayed per page, which columns are shown, metadata about each column if available, whether units and data types are shown at the top of the column, shown here:

By default, it is often but not always the case that all columns are shown. To show or hide columns, select the tickbox in each given row. The default page size is 100 rows. Note that expanding the page size to numbers much greater than 100 may result in a substantial performance degradation (e.g., your browser will appear to freeze or not appear to be doing anything while it manages and renders the large table). See the Filters section below for more things to do from the table options pop-up.

Expand

Clicking on this expands the catalog window pane to take up the entire browser window. To return to the prior view, click on "Close" in the upper left.

Help

The last option on the top of the catalog tab may be a context-sensitive help marker, which should bring you to this online help.

Table Columns

Depending on what you did to display a table, the columns that are shown may be in easily-human-readable form, or may reflect column names used within the individual catalog. Please consult the detailed documentation associated with your specific archive if the headers are not clear to you.

Clicking on the column names sorts the table by that column; clicking once sorts in ascending order, clicking a second time sorts in descending order, and clicking a third time returns the table to the original order. Small arrows appear next to the column names to remind you if the column is sorted in ascending or descending order. When you do a single-position search on catalogs, two new columns are appended to the end of the catalog as it is returned to you. These columns are:

• dist - the distance between the source in question and the location you specified
• angle - the position angle between the target position you requested and the object it found (degrees E of N)

• cntr_01 - the target position you requested
• dist_x - the distance between the target position you requested and the object it found
• pang_x - the position angle between the target position you requested and the object it found (degrees E of N)

Columns in the uploaded table will be included in the output table, with a suffix attached. This suffix will be "_01" unless the column name ends with "_NN" where NN is any integer from 01 to 98. In the latter case, the suffix will be incremented by one, to a maximum value of 99.

⚠ Tips and Troubleshooting

• You can hide or display columns; click on the gears () to get to the table options, and tick the box corresponding to the row you want to hide or show.

Adding Columns

• Name of the column (required) -- it cannot have special characters like a minus sign or a percent symbol; you can only use letters, numbers, and underscores.
• Mode "Enter expression" or "Use preset function" -- options shown here correspond to "Enter expression" options; the preset function options are included in the last bullet here.
• Expression (required) -- using basic SQL operators, you can manipulate columns to create the new column. (See more on this below.)
• Data Type -- specify if your new column is a double precision floating point ("double"), a long integer ("long"), or a string ("char").
• Precision -- if you have selected "double" for data type, select how many decimal places your new column should display. For example, if you want the numbers to display as 1.23, enter "F2".
• Units -- specify the units of your new column. For more information, see IVOA documentation
• UCD, or unified content descriptor -- for VO compliance, add this for your new column. For more information, see IVOA documentation
• Description -- add a description for your new column.
• Select a preset -- if you select "Use preset function", you can choose (a) "set filtered rows to 'true' and the rest to 'false'", (b) "set selected rows to 'true' and the rest to 'false'", or (c) "number rows in current sort order". These options are useful for tagging items you have selected in myriad ways (from plots, images, complicated filters), or ordered in complicated ways. For example, if you have constructed a complicated filter, then you can create a column that is true for the selected rows; if you cancel the complicated filter, you can then easily recreate the complicated filter by simply filtering on your newly created column.

You need to use the column names exactly as they appear in your catalog. Supported operators are: +, -, *, /, =, >, <, >=, <=, !=, LIKE, IN, IS NULL, IS NOT NULL. You may use functions as well; for a list of all available functions, see here . Some examples include:

• "w3mpro" - "w4mpro"
• sqrt(power("w3sigmpro",2) + power("w4sigmpro",2))
• ("ra"-82.0158188)*cos(radians("dec"))
• "phot_g_mean_mag"-(5*log10(1000/"parallax") - 5)

• When you create a new column that is calculated from other columns, it is created statically. That is, it is not dynamically updated like a spreadsheet, but calculated once and left alone after that.
• When you create a new column, the header of the new column is red to let you know that the column is not present in the original catalog.
• When you save the catalog, the header of the saved catalog indicates that you have added a column. When you load the catalog back into the tool, the header of the new column is still red.
• You can edit or delete columns after you have created them; click on the gears () to get to the table options, and then click on the edit icon to bring up a dialog box to edit or delete the column.
• You can hide columns; click on the gears () to get to the table options, and tick the box corresponding to the row you want to hide or show.
• If you create a new column that turns a floating point column into an integer with the "FLOOR()" function, you need to be sure to set the resulting column type to "long". For example, if you have data covering several days or years, and you have a column that is a floating-point MJD, you can convert it into an integer, e.g., via FLOOR(mjd) for days or FLOOR(mjd/365.24) for years, then you can use the drop-down filter menu for the new column to quickly compare different time ranges. But, the new column must be an integer (e.g., "long") in order for this to work properly.
• If you need to, say, take the square root of a column that occasionally has a negative number, and you want it to attempt to handle this in a physically reasonable manner, you can construct expressions like this for a column named 'col' that has some positive and some negative numbers: if("col">=0,sqrt("col"),-sqrt(-"col"))

Table Filters

Example: From a catalog, show only those sources with declination above a certain value (say, 31 degrees), type "> 31" in the box at the top of the "dec" column. Or, if you have retrieved a WISE catalog and would like to only view the objects with a W1 (3.4 micron) profile-fitted magnitude less than 6 magnitudes, in the box at the top of the 'w1mpro' column, type "< 6" in the form.

For columns (fields) with a limited set of choices, on the right edge of the text entry box, an arrow appears; click on it to get a drop-down from which you can select the available choices. To implement the filter, make the choices, and click "Apply" when you are done. Click "clear" in the top of the drop-down menu to remove that filter.

After you impose a filter, then the number of rows in the table is restricted according to the rules you have specified, and the "filters" icon on the top right of the catalog pane has changed to remind you that there has been a filter applied, in this case four filters: . To clear the filters, click on the cancel filters icon (which also appears after you impose filters): .

Filters can be used in combination. Note that the filters between columns are logically "AND"ed together, but filters within the same column can be logically "AND"ed or "OR"ed together; examples are below.

The available logical operators are :

• = which means 'equal to' (exactly!), e.g., the parameter on which you are querying (the column headers as shown) is exactly equal to this value you are specifying.
• > which means 'greater than'
• < which mean 'less than'
• != which means 'not equal to' (exactly!)
• >= which means 'greater than or equal to'
• <= which means 'less than or equal to'
• IN which means 'included within this list', e.g., the parameter on which you are querying is included within the list you are specifying (if the column filter is free-form text, type "value1,value2" and it will give you rows that have value1 or value2).
• LIKE which means 'resembles the text that is entered', e.g., the text resembles the text that you type in the box.
• IS which effectively is the same as =
• IS NOT which effectively is the same as !=

• Retain rows for which a certain parameter is not an empty string: !=''
• Retain rows for which a certain parameter is not NULL and is larger than 1.234: > 1.234 and IS NOT NULL
• Retain rows that have values between -0.5 and 1.25: > -05 and < 1.25
• Retain rows with a parameter greater than one value or exactly not equal to another value: > 12345 or != 3000
• Retain rows with a parameter equal to one of the values in a list: IN a,b,c,d

You may also be able to select rows one at a time via the far left column and then filter that down. Example: Retrieve a catalog of any sort. Select rows by ticking the box on the far left, say, every other row out of the first 12. Click on the filter icon on the top of the column. The filter is imposed to only include the 6 rows you selected.

If you click on the table options icon (), you get a pop-up that includes a place to filter columns. Here is an example of the table columns for a WISE catalog that has been filtered to just have SNR>10 for each of the four bands:

You can type in constraints in the filter box in much the same way as you can from the catalog itself; note that the column description is included here, which may make this way of setting filters more useful when working with a new (to you) catalog.

The second tab is the advanced filter interface:

Here, the columns are listed alphabetically on the left, the constraints you have imposed are in the "current constraints" box, and you can apply additional SQL constraints via the free-form text box. Hints for syntax are given below the entry box. This window can be resized such that you can see the whole set of hints and imposed filters. Note that in this interface, column headers must be enclosed within double quotes.

Cancelling filters

After you impose a filter, then the "filters" icon on the top right of the catalog pane has changed to remind you that there has been a filter applied, in this case just one filter: To clear all the filters at once, click on the cancel filters icon (which also appears after you impose filters): .

⚠ Tips and Troubleshooting

• If you impose logically inconsistent restrictions such as "exposure_time > 160" and "exposure_time < 100" (">160;<100" in the filter box for a column called "exposure_time"), no data will result, because no data are (can be) both less than 160 seconds long and greater than 100 seconds long at the same time. However, "exposure_time > 160 or < 100" works just fine (">160 or <100" in the filter box for the column "exposure-time").
• If you impose nonsensical filters (like using a letter where a number should be, like "w1snr < f") then it will let you know that something has gone really wrong, and let you go back to fix it.
• However, a filter like "ph_qual < f" could be valid -- if the column with which you are working is a string, then a string is a valid filter. It is case-insensitive. For the ph_qual column in the 2MASS catalog, the values are always three letters, such as AAA or ABA or CUU. A filter like "ph_qual < f" will operate as if you have alphabetized the list. Any string that starts with A comes before F and so will be retained. If you do "ph_qual < BBB" then AAA will be left in, but so will "BAU", because alphabetically, BAU precedes BBB.
• If you impose filters from a plot, it can manifest as several filters on the catalog, e.g., one for each side of the square you have drawn on the plot. If you want to remove, say, just one of the four filters (rather than all of them by cancelling all filters), you can do so from the table options pop-up.
• If you want to copy all of the "current constraints", even if the entire field is not visible to you, you can click on the clipboard icon to copy the entire string, and then paste it into another field or application to see what it is.
• If you are choosing filters from a list of terms, cancelling those filters might not work the way it cavalierly seems like it should. If a down arrow appears next to the filter box, then a list of options you can select is available. Tick the boxes you want, and click "Apply" to apply the filter. Now, if you want to change the filter, click the down arrow, select different options, and click "Apply" again to re-impose the new filter. To remove the filter, you have to treat it like you would when applying a modified second filter -- unselect the choices, or hit 'clear', and then hit Apply again. If you deselect the choices and then click elsewhere in the window without hitting clear, your actions are interpreted as 'cancel without doing anything' as opposed to 'impose the new filter I just set' (which is 'cancel all filters'). When you are resetting the filter to be 'select nothing', it is treating that as a new filter setting, so you need to set up that filter and click 'Apply' in order for it to understand.
• If you impose filters from a plot, image, or table, you can cancel them from a plot, image, or table. That is, if you impose filters from a plot, and different additional filters from the image, when you click the 'cancel filters' icon from the table, all the filters are canceled at once. If you have filters imposed from multiple places, clicking on 'cancel filters' doesn't cancel just the filters imposed from that place; it cancels all of them. If you want to be able to reconstruct a complicated set of filters, though, you can add a column to your table that is one of the preset functions -- set it to true if the row is filtered. Then you can cancel all your filters at any time but you can easily recover the filtered data by filtering on that one new column.

Row Details

In some tools, this content appears as additional tab elsewhere in the tool (not in the table pane, but often viewable at the same time as the table itself), as an additional tab called "Details."

In either incarnation, this information is sometimes called a "property sheet."

This table consists of each of the columns of the retrieved catalog with additional information about each field where available. (Not every catalog may have this information available.) For additional information, please consult the full documentation that accompanies the catalog.

Note that if you leave the pop-up or tab open as you select different rows in your main table (or catalog), it is dynamically updated.

⚠ Tips and Troubleshooting

• The property sheet is a more expanded, vertical view of the information shown in a row of a catalog, along with documentation of the catalog columns. If you think of the main table (catalog) view as a single row that you have to scroll left and right to see in its entirety, this view is sort of an orthogonal view, where you have all of the same contents of that row but shown as its own table, with the full header description, if available, and you can scroll up and down to see the entire contents (as opposed to left-right). This is sometimes a more user-friendly way to view any given row.
• Because you can sort/filter the data in the property sheet, you can restrict what values are shown. Those filters are respected as you page through your main table.
• When changing rows in the main table, the property sheet scrolls to preserve the visibility of whatever row in the details tab is highlighted. If you scroll down in the property sheet without changing the highlight, when you change rows in the main table, because the first row in any table is always highlighted by default, the property sheet will scroll back to the top.

Table Cells

If you click on the ellipsis, you get a drop-down menu:
from which you can choose to copy the cell value or view it in a pop-up window.

If you view it in a pop-up window, it will attempt to format it in a readable fashion:

which, for most cells, will be functionally indistinguishable from no special formatting. It will only matter for very complicated cells.

Saving Tables

You obtain this pop-up:

You have several choices to make. In order, they are:

File format

You can save the table in a variety of formats:

It defaults to saving it as an IPAC table file , which is basically ASCII text with headers explaining the type of data in each column, separated by vertical bars.

Other formats include comma-separated values (csv, suitable for, e.g., Excel), tab-separated values (tsv), and three different versions of VO tables . You can also save the file in DS9 Regions file format. See the DS9 website for more information on the syntax of these DS9 region files. The advantage of saving it here (as opposed to from the visualization) is that this way, the entire catalog is guaranteed to be saved.

File name

The tool tries to make a guess at a sensible filename. Feel free to change it to something that makes sense to you.

File location

You may save your file to a local file (on your disk) or, if you are logged in, in the IRSA Workspace .

Modifications to the table

Depending on what you have been doing to the table at this point, you may have filtered or added columns. If you want to save the table as it is currently displayed, with all filters as imposed and any columns hidden, and any added columns as shown, select "Save table as displayed." If you want to save the original table, with all rows and the original columns intact, choose "Save table as originally retrieved."

Table Navigation

There is also another way to navigate among the table tabs.

On the far right, there is a downward pointing arrow. When you click on it, you get a drop-down, showing all the tabs that are loaded. From there, you can select the tab you want to bring to the foreground.