Program Interface: Executing wget and curl commands

The program interface involves constructing URL queries. The commands wget and curl are commonly used for automating these queries. This page provides simple instructions for using these commands to carry out common searches.

In the following examples, you will need to specify the parameters URL and output_filename:


wget

Most IRSA users will execute wget queries with the following syntax. Type "man wget" at the command line to explore the many additional options that wget allows.

  1. To download a file:
  2. % wget "URL"

    This will save the file specified in URL to its original name.

  3. To search around a single position or to download a file and give it your own name:
  4. % wget "URL" -O output_filename
  5. To search around multiple positions (catalogs only):
  6. List a series of query URLs, one for each position, in the file input_filename. Then run the following command: % wget -i input_filename -O output_filename

    For each line in the file input_filename, the program interface will append the returned table to the end of the file output_filename. If the queries return IPAC tables (header rows followed by data rows), you may wish to strip out the header rows for each search result in output_filename in order to make it more easily readable. Using curl avoids this issue and is recommended for multi-object searches.

  7. To search a proprietary data set:
  8. First get a cookie:

    % wget --save-cookies=cookies.txt -O - "https://irsa.ipac.caltech.edu/account/signon/login.do?josso_cmd=login&josso_username=$USER&josso_password=$PASSWORD"

    Use the cookie when doing any requests:

    % wget --load-cookies=cookies.txt "URL"

curl

Most IRSA users will execute cURL queries with the following syntax. Type "man curl" at the command line to explore the many additional options that cURL allows.

  1. To download a file:
  2. % curl -O "URL"

    The -O option tells curl to give the downloaded the same name it has on IRSA's server. Without it, curl will dump the downloaded file into stdout.

  3. To search around a single position or to download a file and give it your own name:
  4. % curl -o output_filename "URL"
  5. To search around multiple positions (catalogs only):
  6. Create an IPAC table "input_filename" containing a list of positions. Then run the following command:

    % curl -F filename=@input_filename / -F spatial=Upload / -F uradius=1 / -F uradunits=arcsec / -F catalog=usno_b1 / -F outfmt=1 / "https://irsa.ipac.caltech.edu/cgi-bin/Gator/nph-query?"
  7. To search a proprietary data set:
  8. First get a cookie:

    % curl -c cookies.txt "https://irsa.ipac.caltech.edu/account/signon/login.do?josso_cmd=login&josso_username=$USER&josso_password=$PASSWORD"

    Use the cookie when doing any requests:

    % curl -b cookies.txt -O "URL"