The Rapthor parset

Before Rapthor can be run, a parset describing the reduction must be made. The parset is a simple text file defining the parameters of a run in a number of sections. For example, a minimal parset for a basic reduction on a single machine could look like the following (see Tips for running Rapthor for tips on setting up an optimal parset):

[global]
dir_working = /path/to/rapthor/working/dir
input_ms = /path/to/input/dir/input.ms

The available options are described below under their respective sections.

Note

An example parset is available in the settings directory.

[global]

dir_working

Full path to working dir where rapthor will run (required). All output will be placed in this directory. E.g., dir_working = /data/rapthor.

input_ms

Full path to the input MS files (required). Wildcards can be used (e.g., input_ms = /path/to/data/*.ms). The paths can also be given as a list (e.g., input_ms = [/path/to/data1/*.ms, /path/to/data2/*.ms, /path/to/data3/obs3.ms]). Note that Rapthor works on a copy of these files and does not modify the originals in any way.

Note

The MS files output by the LINC pipeline can be directly used with Rapthor. See Data preparation for details.

separation_tolerance_arcsec

Sets the pointing separation tolerance (the maximum allowed separation between the pointings; default = 0.05 arcsec). This tolerance is used when more than one observation is included in input_ms to ensure that the observation pointings are consistent with one another.

data_colname

Data column to be read from the input MS files (default = DATA).

generate_initial_skymodel

Generate an initial target sky model from the input data (default = True). This option is ignored if a file is specified with the input_skymodel option. When this option is activated, an image of the full field is made from the input data (without doing any calibration). The initial sky model is generated from the clean components as part of this imaging and will be located in the dir_working/skymodels/initial_image directory.

generate_initial_skymodel_radius

The radius out to which the sky model will be generated (default = None, which results in coverage out to a width of 2 * FWHM of the primary beam at the mean frequency and mean elevation of the observations).

generate_initial_skymodel_data_fraction

Fraction of data to use during the generation of the initial sky model (default = 0.2). If less than one, the input data are divided by time into chunks that sum to the requested fraction, spaced out evenly over the full time range. Very small selections retain at least two time samples when available, as required for imaging, even if this exceeds the requested fraction.

download_initial_skymodel

Download the initial sky model automatically instead of using a user-provided one (default = False). This option is ignored if a file is specified with the input_skymodel option or if generation of the initial model is activated with the generate_initial_skymodel option. The downloaded sky model will be named dir_working/skymodels/initial_skymodel_{catalog}.txt, where {catalog} is the name of the catalog server specified by the download_initial_skymodel_server option.

download_initial_skymodel_radius

The radius in degrees out to which a sky model should be downloaded (default = 5.0).

download_initial_skymodel_server

Place to download the initial sky model from (default = TGSS). This can either be TGSS to use the TFIR GMRT Sky Survey, LOTSS to use the LOFAR Two-metre Sky Survey, or GSM to use the Global Sky Model.

download_overwrite_skymodel

Overwrite any existing sky model with a downloaded one (default = False).

input_skymodel

Full path to the input sky model file, with true-sky fluxes (required if automatic generation or download is disabled). If you also have a sky model with apparent flux densities, specify it with the apparent_skymodel option.

See Data preparation for more info on preparing the sky model.

apparent_skymodel

Full path to the input sky model file, with apparent-sky fluxes (optional). Note that the source names must be identical to those in the input_skymodel.

regroup_input_skymodel

Regroup input skymodel as needed to meet target flux (default = True). If False, the existing patches are used for the calibration.

strategy

Name of processing strategy to use (default = selfcal). A custom strategy can be used by giving instead the full path to the strategy file. See Setting the processing strategy for details on the available predefined strategies and on making a custom strategy file.

selfcal_data_fraction

Fraction of data to use (default = 0.2). If less than one, the input data are divided by time into chunks that sum to the requested fraction, spaced out evenly over the full time range. Using a low value (0.2 or so) is strongly recommended for typical 8-hour, full-bandwidth observations.

final_data_fraction

A final data fraction can be specified (default = 1.0) such that a final processing pass (i.e., after selfcal finishes) is done with a different fraction.

ntimes_to_repeat_final_cycle

Number of times to repeat the final cycle, if done (default = 0). Setting this parameter to a value of one or more may improve the final results but will increase the processing time

input_h5parm

Full path to an H5parm file with direction-dependent solutions (default = None). This file is used if no calibration is to be done.

Note

The time and frequency coverage must be sufficient to cover the duration and bandwidth of the input dataset.

Note

If an imput sky model is also supplied, the directions in the H5parm file must match the patches in the input sky model, or, if not, a facet layout file with matching directions must be supplied and regroup_input_skymodel activated.

input_fulljones_h5parm

Full path to an H5parm file with full-Jones solutions (default = None). This file is used if no calibration is to be done. The notes given for input_h5parm also apply to this file.

input_normalization_h5parm

Full path to an H5parm file with flux-scale normalization solutions (default = None). This file is used if no calibration is to be done. The notes given for input_h5parm also apply to this file.

facet_layout

Full path to a text file that defines the facet layout (default = None). This file must use the WSClean facet format, specified in the WSClean documentation. Also note that the facet centroids (the facet point of interest) must be defined in the file as well. If a facet file is supplied, calibration patches and imaging facets will be set to those specified in the file, if possible, and the calibrator selection parameters specified in the strategy (e.g., target_flux) will be ignored (and therefore the patch and facet layout will be held constant between cycles)

Note

In a given cycle, the calibration patches and imaging facets will match the input facet layout unless the layout would result in one or more empty calibration patches, in which case the empty patches are removed and the layout of the remaining patches is set using Voronoi tessellation.

dde_mode

Mode to use to derive and correct for direction-dependent effects: faceting or hybrid (default = faceting). If faceting, Voronoi faceting is used throughout the processing. If hybrid, faceting is used only during the self calibration steps; in the final cycle (done after self calibration has been completed successfully), IDGCal is used during calibration to generate smooth 2-D screens that are then applied by WSClean in the final imaging step.

[calibration]

use_included_skymodels

Include a packaged sky model in calibration when it is within twice the primary beam FWHM of the field center (default = False).

use_image_based_predict

Use image-based prediction (default = False)? Image-based prediction can be faster than the normal prediction, especially for large sky models.

Note

Currently, correction for time and frequency smearing cannot be done when image-based prediction is used. If the averaging of the input data is such that time or frequency smearing is significant within the field of view of interest, then the use of image-based prediction is not recommended.

use_wsclean_predict

Use image-based prediction using WSClean (default = False)? Usage similar to use_image_based_predict above (but mutually exclusive). Note that the beam application to model data will also be done by this step (DP3 applybeam disabled).

wsclean_predict_bw

Bandwidth (Hz) to create separate images for prediction. The full bandwidth of the data will be divided into chunks of channels whose frequency width given by this value. If the division if not an integer, an approximate (not uniform) division will be made.

wsclean_predict_beam_interval

Time interval (s) to predict facet beam for applying beam in prediction using WSClean.

llssolver

The linear least-squares solver to use (one of qr, svd, or lsmr; default = qr).

maxiter

Maximum number of iterations to perform during calibration (default = 150).

propagatesolutions

Propagate solutions to next time slot as initial guess (default = True)?

solveralgorithm

The algorithm used for solving (one of directionsolve, directioniterative, lbfgs, or hybrid; default = directioniterative). When using lbfgs, the stepsize should be set to a small value like 0.001. For IDGCal solves, the lbfgs algorithm is always used.

onebeamperpatch

Calculate the beam correction once per calibration patch (default = False)? If False, the beam correction is calculated separately for each source in the patch. Setting this to True can speed up calibration and prediction, but can also reduce the quality when the patches are large.

parallelbaselines

Parallelize model calculation over baselines, instead of parallelizing over directions (default = False).

sagecalpredict

Use SAGECal for model calculation, both in predict and calibration (default = False).

fast_datause

This parameter sets the visibilities mode used during the fast-phase solves (one of single, dual, or full; default = single). If set to single, the XX and YY visibilities are averaged together to a single (Stokes I) visibility. If set to dual, only the XX and YY visibilities are used (YX and XY are not used). If set to full, all visibilities are used. Activating the single or dual mode improves the speed of the solves and lowers the memory usage during solving.

Note

Currently, only solveralgorithm = directioniterative is supported when using single or dual modes. If one of these modes is activated and a different solver is specified, the solver will be automatically switched to the directioniterative one.

medium_datause

This parameter sets the visibilities mode used during the medium-phase solves (one of single, dual, or full; default = single). If set to single, the XX and YY visibilities are averaged together to a single (Stokes I) visibility. If set to dual, only the XX and YY visibilities are used (YX and XY are not used). If set to full, all visibilities are used. Activating the single or dual mode improves the speed of the solves and lowers the memory usage during solving.

Note

Currently, only solveralgorithm = directioniterative is supported when using single or dual modes. If one of these modes is activated and a different solver is specified, the solver will be automatically switched to the directioniterative one.

slow_datause

This parameter sets the the visibilities used during the slow-gain solves (one of dual or full; default = dual). If set to dual, only the XX and YY visibilities are used (YX and XY are not used). If set to full, all visibilities are used. Activating the dual mode improves the speed of the solves and lowers the memory usage during solving.

Note

Currently, only solveralgorithm = directioniterative is supported when using the dual mode. If this modes is activated and a different solver is specified, the solver will be automatically switched to the directioniterative one.

stepsize

Size of steps used during calibration (default = 0.02). When using solveralgorithm = lbfgs, the stepsize should be set to a small value like 0.001.

stepsigma

In order to stop solving iterations when no further improvement is seen, the mean of the step reduction is compared to the standard deviation multiplied by stepsigma factor (default = 2.0). If mean of the step reduction is lower than this value (noise dominated), solver iterations are stopped since no possible improvement can be gained.

tolerance

Tolerance used to check convergence during calibration (default = 5e-3).

fast_freqstep_hz

Frequency step used during the fast calibration, in Hz (default = 1e6).

fast_smoothnessconstraint

Smoothness constraint bandwidth used during the fast calibration, in Hz (default = 3e6).

fast_smoothnessreffrequency

Smoothness constraint reference frequency used during the fast calibration, in Hz. If not specified this will automatically be set to 144 MHz for HBA or the midpoint of the frequency coverage for LBA.

fast_smoothnessrefdistance

Smoothness constraint reference distance used during the fast calibration, in m (default = 0).

medium_freqstep_hz

Frequency step used during the medium-fast calibration, in Hz (default = 1e6).

medium_smoothnessconstraint

Smoothness constraint bandwidth used during the medium-fast calibration, in Hz (default = 6e6).

medium_smoothnessreffrequency

Smoothness constraint reference frequency used during the medium-fast calibration, in Hz. If not specified this will automatically be set to 144 MHz for HBA or the midpoint of the frequency coverage for LBA.

medium_smoothnessrefdistance

Smoothness constraint reference distance used during the medium-fast calibration, in m (default = 0).

slow_freqstep_hz

Frequency step used during the slow calibration, in Hz (default = 1e6).

slow_smoothnessconstraint

Smoothness constraint bandwidth used during the slow calibration, in Hz (default = 3e6).

fulljones_freqstep_hz

Frequency step used during the full-Jones calibration, in Hz (default = 1e6).

fulljones_smoothnessconstraint

Smoothness constraint bandwidth used during the full-Jones calibration, in Hz (default = 0).

dd_interval_factor

Maximum factor by which the direction-dependent solution intervals can be increased, so that fainter calibrators get longer intervals (in the fast and slow solves only; default = 1). The value determines the maximum allowed adjustment factor by which the solution intervals are allowed to be increased for faint sources. For a given direction, the adjustment is calculated from the ratio of the apparent flux density of the calibrator to the target flux density of the cycle (set in the strategy) or, if a target flux density is not defined, to that of the faintest calibrator in the sky model. A value of 1 disables the use of direction-dependent solution intervals; a value greater than 1 enables direction-dependent solution intervals.

Note

Direction-dependent solution intervals are not currently supported; they will be re-enabled in a future update.

Note

Currently, only solveralgorithm = directioniterative is supported when using direction-dependent solution intervals. If direction-dependent solution intervals are activated and a different solver is specified, the solver will be automatically switched to the directioniterative one.

dd_smoothness_factor

Maximum factor by which the smoothnessconstraint can be increased, so that fainter calibrators get more smoothing (default = 3). The factors are calculated in the same way as the direction-dependent interval factors, set by dd_interval_factor. A value of 1 disables the use of direction-dependent smoothness factors; a value greater than 1 enables direction-dependent smoothness factors.

solverlbfgs_dof

Degrees of freedom for the LBFGS solver (only used when solveralgorithm = lbfgs; default = 200.0).

solverlbfgs_minibatches

Number of minibatches for the LBFGS solver (only used when solveralgorithm = lbfgs; default = 1).

solverlbfgs_iter

Number of iterations per minibatch in the LBFGS solver (only used when solveralgorithm = lbfgs; default = 4).

average_visibilities

Perform averaging of the input visibilities for imaging (default = True), determined by the maximum allowed before smearing effects become important (see max_peak_smearing).

Note

This works in conjuntion with save_visibilities, so that visibilities used in each imaging cycle can be saved without averaging (unless other averaging such as bda is requested).

bda_timebase

Maximum baseline used in baseline-dependent time averaging (BDA) during the calibration, in m (default = 20000). A value of 0 will disable the averaging. Depending on the solution time step used during the calibration, activating this option may improve the speed of the solve and lower the memory usage during solving.

bda_frequencybase

Maximum baseline used in baseline-dependent frequency averaging (BDA) during the calibration, in m (default = 5000). A value of 0 will disable the averaging. Depending on the solution time step used during the calibration, activating this option may improve the speed of the solve and lower the memory usage during solving.

correct_time_frequency_smearing

Correct for time and frequency smearing during the prediction part of calibration (default = False). Generally, if enabled and imaging is to be done, the identical parameter in the [imaging] section should also be enabled.

[imaging]

cellsize_arcsec

Pixel size in arcsec (default = 1.5).

robust

Briggs robust parameter (default = -0.65).

min_uv_lambda

Minimum uv distance in lambda to use in imaging (default = 80).

max_uv_lambda

Maximum uv distance in lambda to use in imaging (default = 1e6).

mgain

Cleaning gain for major iterations, passed to the imager (default = 0.8). This setting does not affect the first “initial_image” round.

taper_arcsec

Taper to apply when imaging, in arcsec (default = 0).

local_rms_strength

Strength to use for the local RMS thresholding (default = 0.8). The strength is applied by WSClean to the local RMS map using local_rms ^ strength.

local_rms_window

Size of the window (in number of PSFs) to use for the local RMS thresholding (default = 50).

local_rms_method

Method to use for the local RMS thresholding: rms or rms-with-min (default = rms).

do_multiscale_clean

Use multiscale cleaning (default = True)?

bda_timebase (imaging)

Maximum baseline used in baseline-dependent averaging (BDA) in time direction during imaging, in m (default = 20000). A value of 0 will disable the averaging. Activating this option may improve the speed of imaging.

bda_frequencybase (imaging)

Maximum baseline used in baseline-dependent averaging (BDA) in frequency direction during imaging, in m (default = 5000). A value of 0 will disable the averaging. Activating this option should improve the speed of imaging.

dde_method

Method to use to correct for direction-dependent effects during imaging: single or full (default = full). If single, a single, direction-independent solution (i.e., constant across the image sector) will be applied for each sector. In this case, the solution applied is the one in the direction closest to each sector center. If full, the full, direction-dependent solutions are applied (using either facets or screens).

filter_skymodel

Filter out sky model components that lie outside of islands detected by PyBDSF (default = True). If True, only clean components from WSClean whose centers lie inside of detected islands are kept in the sky model used for calibration in the next cycle. If False, all clean components generated by WSClean are kept in the sky model.

Note

It is not recommneded to turn off filtering of the sky model unless the use_image_based_predict parameter is set, as the sky model without filtering can be very large (resulting in runtimes becoming very long unless image-based predict is used).

source_finder

Source finder used when filtering the sky model (default = bdsf).

save_filtered_model_image

Save an image of the filtered sky model during each imaging cycle (default = False).

save_visibilities

Save visibilities used for imaging (default = False). If True, the imaging MS files will be saved, with the the direction-independent full-Jones solutions, if available, applied. Note, however, that the direction-dependent solutions will not be applied unless dde_method = single, in which case the solutions closest to the image centers are used.

save_residual_visibilities

Save the residual visibilities used for the final imaging step (default = False). If True, MS files with the residual visibilities, calculated as data - corrupted model, will be saved in the final cycle. Direction-independent effects, if any, are applied to the data (direction-dependent effects are not, but they are used to corrupt the model data).

Note

The use of this option is not recommended when multiple nights of observations are processed, as the residual dataset is concatenated in time over all observations, with the periods between observations filled with dummy data. This behavior can result in very large residual datasets.

Additionally, if all stations are not present in all obserations, concatenation in time cannot be done, and therefore the saving of residual visibilities is disabled.

save_image_cube

Save frequency cube(s) for the given Stokes parameters (default = False). If True, a cube is constructed from the channel images made during the final imaging step, once self calibration has been completed. The width of the frequency channels in the cube is set by the channel_width_hz parameter in the strategy file. The Stokes parameters for which cubes will be made should be specified with the image_cube_stokes_list parameter.

image_cube_stokes_list

The list of Stokes parameters for which frequency cubes should be saved, specified as a comma-separated list (e.g., [Q, U]; default = [I]).

save_supplementary_images

Save the supplementary images during each imaging cycle (default = False). For now, this is just the PyBDSF-generated masks used for filtering of the sky model (if filter_skymodel = True).

compress_selfcal_images

Compress intermediate selfcal images to reduce storage space (default = True). Uses default fpack compression parameters, see fpack documentation for details on precision. Some tools may be unable to read compressed fits files and will require decompression to be run first. This can be done with the funpack tool .

compress_final_images

Compress the final images to reduce storage space (default = False). See compress_selfcal_images option for compression details.

idg_mode

IDG (image domain gridder) mode to use in WSClean (default = cpu). The mode can be cpu or hybrid.

mem_gb

Maximum memory in GB (per node) to use for WSClean jobs (default = 0 = all available memory).

Note

If the mem_per_node_gb parameter is set, then the maximum memory for WSClean jobs will be set to the smaller of mem_gb and mem_per_node_gb.

apply_diagonal_solutions

Apply separate XX and YY corrections during facet-based imaging (default = True). If False, scalar solutions (the average of the XX and YY solutions) are applied instead. (Separate XX and YY corrections are always applied when using non-facet-based imaging methods.)

make_quv_images

Make Stokes QUV images in addition to the Stokes I image (default = False). If True, Stokes QUV images are made during the final imaging step, once self calibration has been completed.

pol_combine_method

The method used to combine the polarizations during deconvolution can also be specified. This method can be “link” to linked polarization cleaning or “join” to use joined polarization cleaning (default = link). When using linked cleaning, the Stokes I image is used for cleaning and its clean components are subtracted from all polarizations.

disable_iquv_clean

Disable clean for the full-Stokes imaging (default = False). If True, no cleaning is done during full-Stokes imaging.

Note

Clean can be disabled only for full-Stokes imaging (i.e., only when make_quv_images = True).

Warning

Disabling clean can speed up processing, but the resulting images (including Stokes I) may be of poor quality and/or have sources with incorrect flux densities.

dd_psf_grid

The number of direction-dependent PSFs which should be fit horizontally and vertically in the image (default = [0, 0] = scale with the image size, with approximately one PSF per square degree of imaged area). Set to [1, 1] to use a direction-independent PSF.

use_mpi

Use MPI to distribute WSClean jobs over multiple nodes (default = False)? If True and more than one node can be allocated to each WSClean job (i.e., max_nodes / num_images >= 2), then distributed imaging will be used (only available if batch_system = slurm).

Note

If MPI is activated, dir_local (under the [cluster] section below) must not be set unless it is on a shared filesystem.

Note

Currently, Toil does not fully support openmpi. Because of this, imaging can only use the worker nodes, and the master node will be idle.

shared_facet_rw

When using facet-based imaging runs WSClean with the options -shared-facet-reads and -shared-facet-writes (https://wsclean.readthedocs.io/en/latest/facet_based_imaging.html#enabling-shared-reads).

Warning

This option is currently experimental and should be used with caution.

Note

If the number of facets is only one the option will be disabled

reweight

Reweight the visibility data before imaging (default = False). If True, data with high residuals (compared to the predicted model visibilities) are down-weighted. This feature is experimental and should be used with caution.

grid_width_ra_deg

Size of area to image when using a grid (default = 1.7 * mean FWHM of the primary beam).

grid_width_dec_deg

Size of area to image when using a grid (default = 1.7 * mean FWHM of the primary beam).

grid_center_ra

Center of area to image when using a grid (default = phase center).

grid_center_dec

Center of area to image when using a grid (default = phase center).

grid_nsectors_ra

Number of sectors along the RA axis (default = 0). The number of sectors in Dec will be determined automatically to ensure the whole area specified with grid_center_ra, grid_center_dec, grid_width_ra_deg, and grid_width_dec_deg is imaged. Set to 0 to force a single sector for the full area. A grid of sectors can be useful for computers with limited memory but generally will give inferior results compared to an equivalent single sector.

sector_center_ra_list

List of image centers (default = []). Instead of a grid, imaging sectors can be defined individually by specifying their centers and widths.

sector_center_dec_list

List of image centers (default = []).

sector_width_ra_deg_list

List of image widths, in degrees (default = []).

sector_width_dec_deg_list

List of image widths, in degrees (default = []).

max_peak_smearing

Max desired peak flux density reduction at center of the image edges due to bandwidth smearing (at the mean frequency) and time smearing (default = 0.15 = 15% reduction in peak flux). Higher values result in shorter run times but more smearing away from the image centers. Note this option is not considered if average_visibilities = False.

correct_time_frequency_smearing (imaging)

Correct for time and frequency smearing during imaging (default = False). Generally, if enabled and calibration is to be done, the identical parameter in the [calibration] section should also be enabled.

skip_final_major_iteration

Skip the final WSClean major iteration for all but the last processing cycle (default = True). If True, the final iteration is skipped during imaging, which speeds up imaging but degrades the image slightly; however, the sky model is not affected by this setting. Therefore, it is safe to use this option for self calibration cycles.

Note

The final WSClean major iteration is never skipped in the final processing cycle regardless of this setting.

skip_corner_sectors

Skip corner sectors defined by the imaging grid (default = False)? If True and a grid is used (defined by the grid_* parameters above), the four corner sectors are not processed (if possible for the given grid).

use_clean_mask

Use a clean mask during cleaning (default = False)? If True, the clean mask generated by WSClean is used during cleaning. If False, no clean mask is used in wsclean.

photometry_skymodel

Full path to the sky model file for photometry comparison when generating image diagnostics. If this is not set, a sky model will be downloaded from the TGSS and LOTSS catalogs. If this is not possible a backup sky model will be downloaded from the NVSS catalog. Default = None (sky model will be downloaded by default).

astrometry_skymodel

Full path to the sky model file for astrometry comparison when generating image diagnostics. If this is not set, a sky model will be downloaded from Pan-STARRS. Default = None (sky model will be downloaded by default).

normalization_skymodels

List of at least two sky model paths used for flux normalization when enabled by the strategy. The default is None, which allows Rapthor to download suitable sky models when internet access is available.

normalization_reference_frequencies

Reference frequencies, in Hz, corresponding to normalization_skymodels (default = None).

[cluster]

batch_system

Cluster batch system (only used when either StreamFlow or Toil is the CWL runner; default = single_machine). Use single_machine when running on a single machine, slurm to use multiple nodes with dynamic allocation, or slurm_static to use multiple nodes reserved before the run.

Note

When using the slurm batch system, additional Slurm arguments can be passed to Toil by setting the TOIL_SLURM_ARGS environment variable in your environment before running Rapthor. See the Toil environment variables page for details.

max_nodes

When batch_system is slurm or slurm_static, the maximum number of nodes of the cluster to use at once. The configuration default is 0, which is resolved to 1 for single_machine and 12 for Slurm batch systems. When using all the data, each observation is split into up to this many balanced chunks, subject to the minimum calibration duration and two samples per chunk.

cpus_per_task

When batch_system = slurm, the number of processors per task to request (default = 0 = all). By setting this value to the number of processors per node, one can ensure that each task gets the entire node to itself, which is the recommended way of running Rapthor.

mem_per_node_gb

When batch_system = slurm, the amount of memory per node in GB to request (default = 0 = all).

Rapthor also uses this value for DP3 calibration memory checks. A pre-flight check uses the strategy’s maximum number of directions, and a second check before each calibration cycle uses the resolved facet count and DP3 solve intervals. If mem_per_node_gb is zero, the checks compare against memory available on the machine running Rapthor. Slurm users should set mem_per_node_gb because memory available on the machine running Rapthor may not represent memory available on a compute node.

The estimate uses decimal GB and the current DP3 peak-memory model of 80 bytes per visibility sample. The estimate accounts for the original data buffer, visibility copies, weights, and weighted data but excludes the legacy solve buffer. It uses the unaveraged channel count because calibration BDA is baseline-dependent. The result is conservative. By default, Rapthor logs likely out-of-memory configurations but does not change settings or stop processing. Set fail_on_calibration_oom_risk to stop before processing a configuration whose estimate is greater than the applicable limit.

Rapthor calculates the estimated peak memory as follows:

baselines = nstations * (nstations + 1) // 2
time_steps = ceil(solution_interval_seconds / sampling_interval_seconds)
samples = baselines * channels * time_steps * (directions + 1)

visibility_copies_gb = samples * 4 * 8 / 1e9
weights_gb = samples * 4 * 4 / 1e9
weighted_data_gb = samples * 4 * 8 / 1e9
peak_memory_gb = visibility_copies_gb + weights_gb + weighted_data_gb

The baseline count includes autocorrelations. channels is the observation’s unaveraged channel count, and time_steps is rounded up so that a partial solution interval is counted as a full time step. The four in each component is the number of correlations. Complex visibility and weighted-data values use eight bytes per correlation, while weights use four bytes per correlation. Together these components use 80 bytes per sample.

An interactive notebook exploring how estimated peak memory varies with number of baselines, channels, solution interval and directions is available here. (instructions for running it here).

A DI solve always uses one direction. A DD pre-flight estimate uses the strategy step’s max_directions value, while the resolved estimate uses the actual number of calibration facets. The additional direction in directions + 1 accounts for DP3 retaining the original data buffer alongside the direction-dependent data.

fail_on_calibration_oom_risk

Stop Rapthor when a DP3 calibration memory estimate is greater than the applicable memory limit (default = False). This policy is applied both to the pre-flight max_directions upper bound and to the resolved estimate made immediately before each calibration cycle. An estimate exactly equal to the limit is allowed.

When enabled, a high-risk estimate raises an error before the affected pipeline operation starts, and the rapthor command exits with a non-zero status. The error reports the check stage, cycle, solve, observation, estimate, limit, and overage. If memory capacity cannot be determined or the estimate itself cannot be calculated, Rapthor logs an advisory warning and continues because a high OOM risk has not been established.

max_cores

Maximum number of cores per task to use on each node (default = 0 = all).

max_threads

Maximum number of threads per task to use on each node (default = 0 = all).

deconvolution_threads

Number of threads to use by WSClean during deconvolution (default = 0 = 2/5 of max_threads, but not more than 14).

parallel_gridding_tasks

Number of tasks WSClean can use for parallel gridding. If this is set to 0 (default) the number of tasks will be set to max_threads // 8 (most hardware has a multiple of 8 cores per ‘chiplet’ so this is usually a good choice for optimal performance). NB: setting this too low may lead to major thread contention overheads.

dir_local

Full path to a local disk on the nodes for IO-intensive processing (default = not used). The path must exist on all nodes (but does not have to be on a shared filesystem). This parameter is useful if you have a fast local disk (e.g., an SSD) that is not the one used for dir_working. If this parameter is not set, IO-intensive processing (e.g., WSClean) will use a default path in dir_working instead.

Note

This parameter should not be set in the following situations:

• when batch_system = single_machine and multiple imaging sectors are used (as each sector will overwrite files from the other sectors).

• when use_mpi = True under the [imaging] section and dir_local is not on a shared filesystem.

Attention

This parameter is deprecated. Use local_scratch_dir instead.

local_scratch_dir

Full path to a local disk on the nodes for IO-intensive processing (default = None). When batch_system is slurm or slurm_static, the path must exist on all the compute nodes, but not necessarily on the head node. This parameter is useful if you have a fast local disk (e.g., an SSD) that is not the one used for dir_working. If this parameter is not set, IO-intensive processing (e.g., WSClean) will use a default path in dir_working instead.

When cwl_runner = toil and batch_system = single_machine, it is recommended to set this parameter, so that Rapthor can clean up any temporary files and directories that Toil left behind.

Warning

If you want to run multiple instances of Rapthor concurrently using Toil, make sure that you specify different directories as local_scratch_dir. Otherwise, one Rapthor instance will potentially clobber files/directories created by another instance.

global_scratch_dir

Full path to a directory on a shared disk that is readable and writable by all the compute nodes and the head node. This directory will be used to store the intermediate outputs that need to be shared between the different steps in the workflow. If this parameter is not set and batch_system = slurm, then Rapthor will create a temporary directory in dir_working.

When cwl_runner = toil, it is recommended to set this parameter, so that Rapthor can clean up any temporary files and directories that Toil left behind.

Warning

If you want to run multiple instances of Rapthor concurrently using Toil, make sure that you specify different directories as global_scratch_dir. Otherwise, one Rapthor instance will potentially clobber files/directories created by another instance.

use_container

Run the workflows inside a container (default = False)? If True, the CWL workflow for each operation (such as calibrate or image) will be run inside a container. The type of container can be specified with the container_type parameter.

Note

This option should not be used when Rapthor itself is being run inside a container. See Using a (u)Docker/Singularity image for details.

container_type

The type of container to use when use_container = True. The supported types are: docker (the default), udocker, or singularity.

cwl_runner

CWL runner to use. Currently supported runners are: cwltool, streamflow, and toil (default). Toil is the recommended runner, since it provides much more fine-grained control over the execution of a workflow. For example, Toil and StreamFlow can use Slurm to automatically distribute workflow steps over different compute nodes, whereas CWLTool can only execute workflows on a single node. With CWLTool you also run the risk of overloading your machine when too many jobs are run in parallel. For debugging purposes CWLTool outshines Toil, because its logs are easier to understand.

debug_workflow

Debug workflow related issues (default = False). Enabling this option implies that temporary files, produced during the workflow run, will be kept (i.e. the option keep_temporary_files is implicitly set to True). This will require significantly more disk space. The working directory will never be cleaned up, stdout and stderr will not be redirectied, and log level of the CWL runner will be set to DEBUG. Additionally, when using Toil as the CWL runner, some tasks will run using only a single thread (to make debugging easier). Use this option with care!

Note

If Toil is the CWL runner, this option will only work when batch_system = single_machine (the default).

keep_temporary_files

Keep temporary files created during the workflow execution (default = False). If True, temporary files and directories created during the workflow execution will not be deleted at the end of the run. This will require significantly more disk space. This option is useful for debugging purposes.

Note

This option will be set to True automatically when debug_workflow = True.

allow_internet_access

Allow internet access for downloading sky models when these are not provided. Default = True. If False, then the user must either provide the path to a sky model (see input_skymodel) or set generate_initial_skymodel = True. If photometry and/or astrometry skymodels are not provided, then these will not be downloaded and the image diagnostics will not be generated.