Python and CWL code

Rapthor is mainly written in Python and CWL. The Rapthor code tree is organized as follows:

rapthor-master
├── bin
│   └── rapthor
├── docs
├── examples
├── rapthor
│   ├── lib
│   ├── operations
│   ├── pipeline
│   └── scripts
└── test

In the folder structure above:

• rapthor-master/bin contains the rapthor executable used to run Rapthor (see Hardware requirements).

• rapthor-master/docs contains this Sphinx documentation.

• rapthor-master/examples contains an example parset and strategy file.

• rapthor-master/rapthor contains the main Rapthor Python package and CWL files.

• rapthor-master/rapthor/lib contains the main Rapthor classes and modules (see Python classes and modules).

• rapthor-master/rapthor/operations contains the operation subclasses (see Subclasses of the Operation class).

• rapthor-master/rapthor/pipelines contains the CWL pipeline templates (see CWL pipelines).

• rapthor-master/rapthor/scripts contains the processing scripts (see Python processing scripts).

• rapthor-master/test contains files used for testing.

Python classes and modules

The following Python classes and modules are the principal ones used in Rapthor. The corresponding Python files are located in the rapthor/lib directory of the code tree.

• The Operation class
  • Operation
  • Subclasses of the Operation class
• The Observation class
  • Observation
• The Field class
  • Field
• The Sector class
  • Sector
• The cluster module
  • get_available_memory()
  • get_fast_solve_intervals()
  • get_slow_solve_intervals()
• The context module
  • RedirectStdStreams
  • Timer
• The miscellaneous module
  • approx_equal()
  • create_directory()
  • dec2ddmmss()
  • delete_directory()
  • download_skymodel()
  • get_flagged_solution_fraction()
  • get_reference_station()
  • make_template_image()
  • normalize_dec()
  • normalize_ra()
  • ra2hhmmss()
  • rasterize()
  • read_vertices()
  • remove_soltabs()
  • string2bool()
  • string2list()
• The parset module
  • get_calibration_options()
  • get_cluster_options()
  • get_global_options()
  • get_imaging_options()
  • parset_read()

Python processing scripts

The Rapthor pipelines call a number of Python scripts to process the solutions, images, etc. The scripts are located in the rapthor/scripts/ directory of the code tree. For details of each script’s function, see the inline documentation in the script’s code. A description of the inputs can also be obtained by running the script with the -h flag.

CWL pipelines

The CWL pipeline parsets and step definition files are located in the rapthor/pipeline/ directory of the code tree. Each operation in Rapthor has a corresponding pipeline parset. An overview of each operation is given in Operations. For details of each step of the pipelines, see the inline documentation in the pipeline parset files (in rapthor/pipeline/parsets) and step files (in rapthor/pipeline/steps).

Note

The CWL parset files in rapthor/pipeline/parsets are jinja2 templates, and so are not directly parsable by CWL tools. Rapthor uses the templates to generate the actual CWL parsets that are passed to Toil. These generated files are created in the Rapthor working directory in workdir/pipelines/operation_name at runtime with the following names: the pipeline CWL parset file is named pipeline_parset.cwl (and subpipeline_parset.cwl when there is a subpipeline) and the pipeline inputs JSON file is named pipeline_inputs.json.