Program Interface: Executing wget and curl commands

The program interface involves constructing URL queries. The commands wget and curl are commonly used for automating these queries. This page provides simple instructions for using these commands to carry out common searches.

In the following examples, you will need to specify the parameters URL and output_filename:

• URL refers to the string that describes the search that you want an IRSA Service to carry out, or the file that you wish to download. The format of this string depends on the type of data you are interested in.


• output_file specifies the name of the file that will contain the results of the query. Unless otherwise specified in the URL, this will be in a default format that depends upon the service being queried.

---

wget

Most IRSA users will execute wget queries with the following syntax. Type "man wget" at the command line to explore the many additional options that wget allows.

1. To download a file:
% wget "URL"

This will save the file specified in URL to its original name.

2. To search around a single position or to download a file and give it your own name:
% wget "URL" -O output_filename
3. To search around multiple positions (catalogs only):

List a series of query URLs, one for each position, in the file input_filename. Then run the following command: % wget -i input_filename -O output_filename

For each line in the file input_filename, the program interface will append the returned table to the end of the file output_filename. If the queries return IPAC tables (header rows followed by data rows), you may wish to strip out the header rows for each search result in output_filename in order to make it more easily readable. Using curl avoids this issue and is recommended for multi-object searches.

4. To search a proprietary data set:

First get a cookie:

% wget --save-cookies=cookies.txt -O - "https://irsa.ipac.caltech.edu/account/signon/login.do?josso_cmd=login&josso_username=$USER&josso_password=$PASSWORD"

Use the cookie when doing any requests:

% wget --load-cookies=cookies.txt "URL"

---

curl

Most IRSA users will execute cURL queries with the following syntax. Type "man curl" at the command line to explore the many additional options that cURL allows.

1. To download a file:
% curl -O "URL"

The -O option tells curl to give the downloaded the same name it has on IRSA's server. Without it, curl will dump the downloaded file into stdout.

2. To search around a single position or to download a file and give it your own name:
% curl -o output_filename "URL"
3. To search around multiple positions (catalogs only):

Create an IPAC table "input_filename" containing a list of positions. Then run the following command:

% curl -F filename=@input_filename / -F spatial=Upload / -F uradius=1 / -F uradunits=arcsec / -F catalog=usno_b1 / -F outfmt=1 / "https://irsa.ipac.caltech.edu/cgi-bin/Gator/nph-query?"
4. To search a proprietary data set:

First get a cookie:

% curl -c cookies.txt "https://irsa.ipac.caltech.edu/account/signon/login.do?josso_cmd=login&josso_username=$USER&josso_password=$PASSWORD"

Use the cookie when doing any requests:

% curl -b cookies.txt -O "URL"