Knowledgebase: MOPEX and APEX

MOPEX is damaged and can't be opened.

For Mac OSX versions starting with Lion and some settings of Gatekeeper, you may receive the following error message when you attempt to run MOPEX:

        "MOPEX" is damaged and can't be opened. You should move it to the Trash.
Click "Cancel". You will need to temporarily disable Gatekeeper, launch MOPEX, and then re-enable Gatekeeper again (if you wish). You only need to do this once. Afterwards, MOPEX will launch in the usual manner.

To temporarily disable Gatekeeper, open System Preferences. Under "Personal", click on "Security & Privacy". If necessary, click the lock and enter your password. Under "Allow applications downloaded from:", select the least restrictive option. You may leave this window open.

Go back to the finder and double-click on the spice application to launch it. You will receive the following message:

        "MOPEX" is an application downloaded from the Internet. 
        Are you sure you want to open it?
Click "Open".

Now you can go back to the "Security & Privacy" window and set the "Allow applications downloaded from:" to whatever value you are comfortable with.

Whenever you wish to use MOPEX in the future, you can just double click without altering Gatekeeper.

What MOPEX namelist should I use for crowded fields?

We don't currently have any special MOPEX/APEX namelists for crowded fields. Many people have used the default namelists. If using the GUI, see Cookbook Recipe 7 And the command-line namelists are identical.

There are 2 things to consider. One is whether the Mosaic task has done a good job. Check whether there are uncorrected radhits on the mosaic. The module "box_outlier" is on by default and designed to work with coverage as low as 3. If needed, it could be made more aggressive by lowering the "BOX_THRESHOLD" parameters in the MosaicRmask block.

Second is whether Detect is optimized for your mosaic. The default namelists should be okay for the bulk of the field. You might do a second special run for a crowded area -- the main parameter in Detect is Detection_Threshold which you could raise for the crowded area. You could also raise Min_Coverage if radhits are slipping in.

But there will always be some fields that are just too difficult for an automated program to do properly.

Should I be using the MOPEX GUI or the command-line version?

The command-line and GUI versions of MOPEX (including APEX) are identical in terms of the results they produce. If you are a new user then we strongly recommend using the GUI, as it is more user-friendly and has enhanced, built-in help pages. If you are running an operating platform other than Solaris 10+, Mac OSX 10.3+ or Linux RedHat then we also recommend using the GUI, as many users have experienced Perl library linking problems while running the command-line version on other platforms. If you are a seasoned MOPEX user, and/or want to run large batch jobs then you may want to consider using the command-line version, as it is marginally faster and can be scripted. Having said that, even experienced users may find the GUI help pages useful.

How do I get hold of the command-line version of MOPEX (including APEX)?

The command-line version of MOPEX is distributed with the GUI - you should download the GUI and follow the installation instructions on the MOPEX Download page. Note that the environment variable file for MOPEX has changed from mopex.csh in previous versions to mopex-script-env.csh.

I'm using a flavor of Linux other than RedHat (e.g. Debian / Fedora / Ubuntu) and I'm getting strange error messages when I try to run the command-line version of MOPEX (including APEX)

The command-line version of MOPEX is only supported on the following platforms: Solaris 10+, Mac OSX 10.3+ and Linux RedHat. If you are running a different platform or flavor of Linux then you are probably experiencing Perl library linking problems (although it may not be obvious from the error messages you are seeing). You should try upgrading to the latest release of MOPEX, but if that doesn't work then you should try using the MOPEX GUI. If you are getting glibc errors then see the MOPEX Bug List for a possible fix.

I'm trying to run the MOPEX GUI on my Linux box but it fails with an error message about initialization of the VM

Some Linux users have seen the following error message when trying to start MOPEX:

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

The MOPEX GUI startup settings are too aggressive for your machine. You need to edit the following line in the executable file <installation_dir>/mopex:

Replace the following line:
set MEM = (-Xms48m -Xmx2048m)

with this:
set MEM = (-Xms48m -Xmx1024m)

then run the script to start MOPEX. MOPEX will prompt you to auto-update. Select "Quit and Install" to update to the latest version. After MOPEX is updated you must repeat the above steps as the executable will have been overwritten. You should then be able to start the MOPEX GUI without a problem.

What do I need to start using MOPEX?

If you are using the GUI then all you need is a set of input images listed in a text file.

If you are using the command line, you need a set of input images with their full path name listed in a text file, and you will also require a "namelist" - a file including all the input parameters for running MOPEX.

There are a number of other inputs that are usually included - see the MOPEX User's Guide for more details.

Are there any restrictions on the size of the mosaic created by MOPEX due to the projection approximations?

The projection used here is exact, it involves no approximations. The only requirement here is that the final mosaic can not span more than one hemisphere.

What kinds of images can be processed by MOPEX?

Any images in the FITS format, but the following keywords are required:
BITPIX, NAXIS, NAXIS1, NAXIS2, CRVAL1, CRVAL2, CRPIX1, CRPIX2, CTYPE1, CTYPE2.
Either the CD matrix or CDELT1, CDELT2, CROTA2 keywords are also required.

Are there any restrictions on the mosaic pixel size in MOPEX?

There are no restrictions on the mosaic pixel size. If the mosaic pixel size is so small that you don't have enough computer memory to coadd all the interpolated images at once, you should split the mosaic into several tiles. In order to do so, a suggested tile size is specified in the namelist:

&MOSAICCOADDIN
TILEMAX_X = 1000,
TILEMAX_Y = 1000,
&END

See the MOPEX User's Guide for more details.

How do I decide which interpolation method in MOPEX to use to create my science mosaic?

There are 4 interpolation methods available in MOPEX: DEFAULT (1), DRIZZLE (2), GRID (3), and CUBIC (4).

GRID should be used when a quick-look mosaic is needed. It can also be used for background matching (overlap.pl). The gain in speed is up to 10 times as fast as DEFAULT, without much loss in the quality of the interpolated image. The price you pay is the fidelity of the interpolated images. It should not be used for photometry purposes. It should also not be used while doing outlier rejection. If you choose to use GRID, don't forget to set GRID_RATIO in the namelist.

The main differences between the remaining choices are in how the noise properties will be correlated in the final mosaic and in the image quality. Most users will choose between DEFAULT and DRIZZLE. The main advantages of DRIZZLE are reduced noise correlation and marginally improved image quality. The main disadvantages are non-uniform coverage and noise properties which can be difficult to model. CUBIC interpolation could theoretically improve image quality in cases of high S/N because it enforces smoothness in the interpolation function. However, CUBIC interpolation will also produce significantly more noise correlation than a linear interpolation (used in the DEFAULT mode).

A rule of thumb is that if you have a coverage of 10 or less, you will probably want to use the DEFAULT interpolation. The DEFAULT interpolation method has one optional input parameter: Fine Res (FINERES in namelists). This should be set to zero, if present at all, as this feature is not fully implemented. If you have a coverage of 10 or more, you may want to consider the use of DRIZZLE. The only input parameter for this is the Drizzle Factor (DRIZ_FAC in namelists). This is the factor by which each input pixel is shrunk before it is projected onto the final image. The smaller you make the Drizzle Factor, the more coverage you need. If you use DRIZZLE with a small Drizzle Factor and/or low coverage, you could produce holes in your mosaic. Even if you don't end up with holes, the final pixels are more likely to be affected by outliers because fewer data pixels go into each mosaic pixel. The coverage (and hence effective gain) of the mosaic could show significant variations from pixel to pixel.

More information on image interpolation can be found in the MOPEX User's Guide.

Which MOPEX modules should be turned on in the namelist for various outlier rejection options?

For Single Frame Outlier Rejection, the following should be included (set to 1 in the namelist):
MedFilter (run_medfilter = 1)
Detect RadHit (run_detect_radhit = 1)

For Multiframe Temporal Outlier Rejection the following should be included (set to 1 in the namelist):
Mosaic Outlier (run_mosaic_outlier = 1)
Check the "Use Outlier for Rmask" box in the Mosaic RMask module (USE_OUTLIER_FOR_RMASK = 1)

For Dual Outlier Rejection the following should be included (set to 1 in the namelist):
MedFilter (run_medfilter = 1)
Detect Outlier (run_detect_outlier = 1)
Mosaic Projection (run_mosaic_proj = 1)
Mosaic Dual Outlier (run_mosaic_dual_outlier = 1)
Level (run_level = 1)
Check the "Use Dual Outlier for Rmask" box in the Mosaic RMask module (USE_DUAL_OUTLIER_FOR_RMASK = 1)

For Box Outlier Rejection, the following should be included (set to 1 in the namelist):
Mosaic Box Outlier (run_mosaic_box_outlier = 1)
Check the "Use Box Outlier for Rmask" box in the Mosaic Rmask module (USE_BOX_OUTLIER_FOR_RMASK = 1)

Interpolated images are required for multiframe and dual outlier rejection. The following module should be run once before running Mosaic RMask:
Mosaic Coverage (run_mosaic_covg = 1)

Note that you also need to ensure that your RMask Fatal Bit Pattern is set to use the results from your outlier rejection method. This is set in the Initial Settings module in the GUI, under Optional Input and Mask Files. In the namelist the Fatal Bit Pattern is set with the keyword
RMask_Fatal_BitPattern = <desired fatal bit pattern>

See the MOPEX User's Guide for more information on outlier rejection schemes and fatal bit patterns.

When running the MOPEX Box Outlier Rejection scheme, the results of the outlier rejection aren't included in the RMask. What's going on?

There are 3 steps required to use the results of Box Outlier Rejection in your reduction: 1) add the Mosaic Box Outlier module to your processing flow (run_mosaic_box_outlier = 1); 2) in the Mosaic RMask module settings, click the check box for "Use Box Outlier" (USE_BOX_OUTLIER_FOR_RMASK = 1); 3) in the Initial Settings module, set the Rmask Fatal Bit Pattern to include the results of Box Outlier Rejection (RMask_Fatal_BitPattern = 3 to use Box Outlier only. See the MOPEX User's Guide for more information on how to combine the bit patterns for multiple outlier rejection schemes).

How do I set the pixel size of the mosaic in MOPEX?

The pixel size can be set directly using the following keywords in the namelist, or in the Initial Settings module General Options in the GUI:

MOSAIC_PIXEL_SIZE_X = -0.0003
MOSAIC_PIXEL_SIZE_Y = 0.0003

MOSAIC_PIXEL_SIZE_X is negative to follow the convention that CDELT1 is negative.

Alternatively, you can set the ratio of the input image pixel size to the mosaic pixel size using the following keywords:

MOSAIC_PIXEL_RATIO_X = 2
MOSAIC_PIXEL_RATIO_Y = 2

I'm getting a MALLOC_FAILED error message when I run mosaic.pl. Help!

I'm getting the following error message when I run mosaic.pl:

Pipeline Module MOSAIC_COADD Version 2.5
Processing time Tue Jan 20 18:11:26 2004
nmings 10
Error in allocating memory for Coverage Image for Tile 0
mosaic_coadd Error Message 0x0071: MALLOC_FAILED.

This is an indication that you are out of RAM. The chances are that you are trying to make a very big mosaic. First, make sure that this is what you actually want to do. Check the keywords NAXIS1, NAXIS2 in the Fiducial Image Frame file in your output directory (FIF.tbl). It is possible that a huge FIF was created because one or more images in the set have bad pointing. Verify that all the images have reasonable pointing. (headerlist.tbl). If you believe that the problem is real, i.e. you are actually trying to make a mosaic as big as it is in FIF.tbl, then use the tiling capabilities by specifying the tile size for the appropriate module in the namelist. For Mosaic Coadd (mosaic_coadd) and Mosaic Coverage (mosaic_covg) the keywords are:

TILEMAX_X = 1000,
TILEMAX_Y = 1000,

For Mosaic Outlier (mosaic_outlier) and Mosaic Dual Outlier (mosaic_dual_outlier) the keywords are:

TILE_XSIZ = 500,
TILE_YSIZ = 500,

See the MOPEX User's Guide for more information.

I have a list of images covering a big area on the sky, but I only want MOPEX to make a mosaic covering a small portion of that area. How can I do it?

1) Run Mosaic with only the Fiducial Image Frame module included in the flow (set run_fiducial_image_frame = 1 if using the command line) to create a Fiducial Image Frame table that includes all of the images in the entire input list.

2) Edit the resultant FIF.tbl (found in the output directory) to cover only the spatial area that you are interested in.

3) Run Mosaic again, but with the following settings:
Include the Mosaic Geometry module (run_mosaic_geom = 1)
Take out the Fiducial Image Frame module (run_fiducial_image_frame = 0).
Specify your modified FIF.tbl file as input in the Initial Settings > Optional Input & Mask Files > FIF file (FIF_FILE_NAME = FIF.tbl)

Mosaic Geometry will identify all of the input files that cross your modified FIF and will only use those files for the rest of the mosaicking process.

After I have run overlap.pl the mosaic doesn't have the patchiness, but it now displays an overall gradient. Is it real or was it introduced by overlap.pl?

The chances are that it's neither of the above. Most likely the overall gradient in the mosaic image comes from the gradients in the input images, which were not properly flat-fielded. This is sometimes seen in MIPS-24 images. Perl script flatfield.pl, included in the MOPEX distribution, should take care of this problem, or you can carry out your own flatfielding of the input images with your favorite software. Once the images are properly flat-fielded, you can re-run them through MOPEX to create your mosaic.

I have run MOPEX medfilter to create background-subtracted images for overlap.pl. It took me forever because I used a big window size. I don't want to do it again for mosaic.pl and apex.pl. Is there an easy way to re-use those background-subtracted images

Yes. If a subdirectory name is specified as a full path, it is not treated as a subdirectory of OUTPUT_DIR and the name is used as specified. All you have to do is specify MEDFILTER_DIR to point to the location where the background-subtracted images created by overlap.pl were saved. For example, let's say that in the overlap namelist you have:
OUTPUT_DIR = output_overlap
MEDFILTER_DIR = Medfilter

and you are running overlap.pl and mosaic.pl in the same directory /home/NGC0001/.

In order to re-use the background-subtracted images, you would write in the mosaic namelist:
MEDFILTER_DIR = /home/NGC0001/output_overlap/Medfilter

If you are using the GUI, the output directories can be set in the individual module settings.

The MOPEX module prf_estimate is failing with a Perl error, or with an error about incorrect "flux" columns in a PRF file. What's wrong?

The Perl error is most likely a memory limit because you are trying to derive a PRF from too many stars x too many frames x too big a fitting area x too fine sampling. For example, if you have a big fitting area, you could reduce this by lowering PostStamp_Xsize and PostStamp_Ysize.

The second error is due to MOPEX having trouble with the flux column in the input table. This is a bug that remains to be fixed. You can get around it by having a table with just RA and Dec, or RA and Dec and X and Y. It will then determine an approximate flux for each PRF star for its fitting.

I have made a mosaic using mosaic.pl. Do I have to redo it when I run apex.pl?

No, you don't have to redo mosaicking, especially if outlier rejection is used as part of mosaicking, since APEX does not do outlier rejection. APEX can use the tiles created by Mosaic and saved in the coadd directory (COADDER_DIR). If you are using the GUI then you should Insert the APEX pipeline after the Mosaic pipeline, and remove the Fiducial Image Frame, Mosaic Interpolate, Mosaic Coadder and Mosaic Combine modules from the APEX flow. If you are using the command line then the following option should be set in the mosaic namelist:

keep_coadded_tiles = 1

Then set OUTPUT_DIR to be the same in both the apex.pl and mosaic.pl namelists. When running apex.pl, the run_fiducial_image_frame, run_mosaic_interp, run_mosaic_coadder and run_mosaic_combiner steps can and should be skipped (set to 0 in the APEX namelist).

The detection list generated by MOPEX - mosaic_detect.tbl - has no detections.

Parameter Detection_Threshold is probably set too high in the Detect module. Try lowering it. If this doesn't help, check that your Min_Coverage parameter isn't higher than the actual coverage in your mosaic.

I don't want to do MOPEX passive deblending. Is there a way to turn it off?

The way to avoid doing passive deblending is to remove the deblend_size and deblend_id columns. To do so on the command line, set the following in the namelist:
run_select_detect = 1

and add the list of columns to copy:
select_detect_columns = "srcid,x,y"

In the GUI, you should add the module Select Detect into the flow, and add the columns srcid, x, y in the module settings.

Select Detect will create a new detection table without the deblend_size and deblend_id colummns, and passive deblending will not be performed.

The values of chi2/dof's in the extraction table generated by MOPEX - extract.tbl - are way too high.

Check the uncertainty images - they could be underestimated.

All of the chi2/dof's in the extraction table generated by MOPEX - extract.tbl - are ~0.1 or lower.

Check the uncertainty images - they could be overestimated.

Can I do point source extraction on the MOPEX mosaicked image?

Yes. First you should produce a mosaic with the corresponding uncertainty image. Then you run apex_1frame.pl (APEX Single Frame in the GUI) setting INPUT_IMAGE_FILE = <mosaic_image_file_name> and SIGMA_IMAGE_FILE = <mosaic_uncertainty_image_file_name>. Use the appropriate PRF file. PRF fitting is not recommended on IRAC mosaics - either use aperture photometry or do the PRF-fitting on the stack of BCDs with apex.pl

How do I get APEX to perform aperture photometry for a given set of sources in the sky?

This is possible with the APEX "User List" flows. See the MOPEX User's Guide for more information.

My apex.pl run crashed with an obscure message about code 6. What's the problem?

The error message:

Perl system() return code = 6
sourcestimate ran with signal 6
System Exit Code ( sourcestimate): 6

usually means that you are out of RAM because of big blends of detections in the detection table. The module Source Estimate performs simultaneous fitting of all the detections in the detection table with the same value of the blend_id column (passive deblending). In order to bypass this problem, you need to clean the detection table by removing huge "blends" of detections. You can do it on the command line by inserting the following two lines in the namelist:
run_select_detect = 1
select_detect_conditions = "blend_size < 100"

In the GUI you should add the Select Detect module into the flow and add a condition specifying a maximum blend size.

The actual value for the blend_size limit may vary with the data set. In order to find the value applicable for your data you can run select directly on the detect table, by typing the following on the command line:

select from mosaic_detect.tbl for "blend_size > 100"

Select will print out the number of rows with blend_size > 100. If there are none then you can decrease the limit.

How do I modify my FIF.tbl so that MOPEX makes my output mosaic the size of my choice?

Here is an example FIF.tbl file, which is by default placed in your output directory by the MOPEX module Fiducial Image Frame.

********************************************************************************
\char comment = Output from fiducial_image_frame, version 1.4
\char Date-Time = Jan 28, 2009, 10:13:05
\real CRVAL1 = 217.950767
\real CRVAL2 = 34.074341
\real CRPIX1 = 2455.50
\real CRPIX2 = 2900.00
\real CROTA2 = 0.00000
\real CDELT1 = -6.92842E-04
\real CDELT2 = 7.21978E-04
\int NAXIS1 = 4910
\int NAXIS2 = 5799
\int EQUINOX = 2000
\char PROJTYPE = TAN
\char COORDINATE_SYSTEM = J2000
\char CTYPE1 = RA---TAN
\char CTYPE2 = DEC--TAN
\real EXTENT_X = 3.401852
\real EXTENT_Y = 4.186750
********************************************************************************

Let's say you wish to edit this FIF.tbl so that your mosaic ends up centered at RA = 218 degrees, DEC = 34 degrees, and you want your mosaic to be 1 degree on a side.

Then you would change the following values in your FIF.tbl :

CRVAL1 = desired reference RA of output mosaic in degrees = 218

CRVAL2 = desired reference DEC of output mosaic in degrees = 34

NAXIS1 = desired length of output rectangular region in pixels = | 1 degree / CDELT1 | = 1443

NAXIS2 = desired width of output rectangular region in pixels = | 1 degree / CDELT2 | = 1385

CRPIX1 = pixel coordinate corresponding to CRVAL1, usually the center of the mosaic
= ( NAXIS1 + 1 ) / 2 = 722

CRPIX2 = pixel coordinate corresponding to CRVAL2, usually the center of the mosaic
= ( NAXIS2 + 1 ) / 2 = 693

"You can't open the application "mopex" because PowerPC applications are no longer supported" error when opening MOPEX on MacOSX

More recent versions Mac OS (>10.9) see that a PowerPC executable is bundled in the MOPEX package and prevents the package from being run by clicking on the Application in a Finder window or the dock. You can still launch MOPEX from the command line using:

 /Applications/mopex/mopex.app/Contents/MacOS/JavaApplicationStub &