Pipeline configuration reference#

This page summarises the YAML keys understood by MoDaCor pipeline definitions.

Step fields#

Each entry under steps is keyed by a step_id and supports the following fields:

  • module (required): ProcessStep class name to instantiate.

  • requires_steps (optional): list of step_ids that must run before this step.

  • configuration (optional): dictionary of ProcessStep configuration values.

  • short_title (optional): a brief, human-friendly purpose label used in graphs (Mermaid/DOT). This is appended as a second line in node labels, e.g. AU: MultiplyDatabundles + scaling to absolute units.

Repeated step blocks#

The optional top-level step_blocks mapping compactly describes repeated single steps or subpipelines. Every block has:

  • for_each: item id to parameter mapping;

  • steps: one or more ordinary step templates keyed by local step id.

step_blocks:
  prepare:
    for_each:
      sample:
        processing_key: sample
        enabled: true
      background:
        processing_key: background
        enabled: false
    steps:
      first:
        module: SomeStep
        configuration:
          with_processing_keys: ["${processing_key}"]
          enabled: "${enabled}"
      second:
        module: AnotherStep
        requires_steps: [.first]
        configuration:
          with_processing_keys: ["${processing_key}"]

Expansion is deterministic. Generated ids have the form <block>.<item>.<local-step>. A prerequisite beginning with . resolves to a step in the same item; other prerequisite ids are absolute. Ordinary top-level steps may depend on generated ids, and block steps may depend on ordinary steps.

Dependencies on an expanded block#

A block is an authoring construct, not an execution node. Consequently, requires_steps: [prepare] does not mean “wait for the prepare block” and is invalid unless an ordinary step named prepare exists. There is no implicit block-wide dependency or wildcard in version 1.

Downstream steps name the generated instances they actually require. Usually these are the terminal steps of the relevant item chains:

steps:
  combine:
    module: CombinePreparedData
    requires_steps:
      - prepare.sample.second
      - prepare.background.second

Because prerequisites are transitive, the downstream step does not also need to list prepare.sample.first when prepare.sample.second already depends on it. If it needs only one lane, it should require only that lane’s generated terminal step. Within a block template, use the local form .first; outside the block, use the complete <block>.<item>.<local-step> id.

A scalar that consists entirely of ${parameter} is replaced recursively by the native parameter value. Consequently booleans, numbers, lists, and mappings remain typed. Partial string interpolation such as prefix-${parameter} is not supported; provide the complete string as an item parameter instead.

Block, item, and local-step ids contain letters, digits, underscores, or hyphens. Version 1 deliberately excludes nested blocks, conditional steps, Cartesian products, and expression evaluation. Use another block or an explicit ordinary step for exceptional lanes.

Expansion happens before ProcessStep instantiation. The scheduler therefore receives the same flat DAG as an explicitly written pipeline. Exported expanded YAML contains ordinary steps only, while graph specs attach origin metadata to generated nodes for grouping and diagnostics. Pipeline.to_dot() and Pipeline.to_mermaid() use that metadata to group blocks and item lanes by default; pass group_step_blocks=False to either method for the flat execution graph.

Step configuration validation#

Each configuration block is checked when Pipeline.from_yaml(...) loads the pipeline. MoDaCor builds the accepted key list and top-level type policy from the selected module class:

  • shared ProcessStep keys from CONFIG_KEYS

  • module-specific keys from ProcessStepDescriber.arguments

For example, Divide.divisor_source is declared as a string source reference, so this is valid:

steps:
  normalize:
    module: Divide
    configuration:
      with_processing_keys:
        - sample
      divisor_source: sample::entry/frame_exposure_time

and this fails during pipeline loading because divisor_source is not a string:

steps:
  normalize:
    module: Divide
    configuration:
      divisor_source: 3

The central validator catches unknown keys and top-level type mismatches. Module code still performs semantic checks for values that need runtime context, such as missing sources, non-empty required strings, mutually exclusive options, or nested dictionary contents.

DataBundle arithmetic#

The source-based Divide, Subtract, and Multiply modules load their second operand from IoSources. Their interfaces remain source-oriented.

Use DivideDatabundles when both operands have already been prepared as BaseData entries in ProcessingData. with_processing_keys contains exactly two keys: the dividend first and the divisor second. The default entry name is signal in both bundles; dividend_data_key and divisor_data_key can select different entries.

steps:
  normalize_to_count_time:
    module: DivideDatabundles
    configuration:
      with_processing_keys: [sample, sample_count_time]
      dividend_data_key: signal
      divisor_data_key: signal

The dividend is updated in place and returned. BaseData arithmetic supplies array broadcasting, unit calculation, and uncertainty propagation. This is the division counterpart of the existing SubtractDatabundles and MultiplyDatabundles steps.

Sampled-data integration#

Integrate1D integrates one or more sampled curves with trapezoidal or Simpson quadrature. The curves share a coordinate array, which may be nonuniform but must be strictly monotonic. Coordinate units are multiplied into the result units and each uncertainty component is propagated independently with the quadrature coefficients.

steps:
  integrate_sample_and_blank:
    module: Integrate1D
    configuration:
      with_processing_keys: [sample_curve, blank_curve]
      signal_key: signal
      axis_key: q
      method: trapezoid
      output_processing_keys: [sample_integral, blank_integral]

Invalid, masked, or zero-weight samples in any input are omitted from every integral so ratios use the same physical domain. Restricting this module to 1D keeps the coordinate and uncertainty rules unambiguous; an n-dimensional sampled-data integrator can be introduced later if a concrete pipeline requires one.

NeXus detector frames#

MoDaCor has a generic NeXus transformation-chain resolver in the base modules. It follows scalar depends_on links, resolves relative NeXus paths, and returns a 4x4 affine transform matrix in SI length units. The resolver is deliberately not a scattering-only feature: the same NeXus transformation-chain convention can describe detectors, sample stages, and other instrument components.

Scattering detector coordinate modules can use this resolver through a detector_frame configuration block. This keeps pipeline YAML lean because PixelCoordinates3D can read the detector origin, fast/slow pixel directions, pixel pitches, and detector normal from the configured NXdetector directly:

steps:
  pixel_coordinates:
    module: PixelCoordinates3D
    configuration:
      with_processing_keys: [static]
      detector_frame:
        type: nexus
        source: calibration
        detector_path: /entry1/instrument/detector
        module_origin: first_pixel_center

XSGeometryFromPixelCoordinates accepts the same detector_frame block to reuse the NeXus pixel pitches and detector normal for solid-angle calculation:

steps:
  scattering_geometry:
    module: XSGeometryFromPixelCoordinates
    requires_steps: [pixel_coordinates]
    configuration:
      with_processing_keys: [static]
      detector_frame:
        type: nexus
        source: calibration
        detector_path: /entry1/instrument/detector
        module_origin: first_pixel_center
      sample_z_override:
        value: 0.0
        units: mm
      photon_source: calibration::/entry1/calibration_sample/beam/incident_wavelength
      photon_units_source: calibration::/entry1/calibration_sample/beam/incident_wavelength@units

photon_source may contain photon energy or wavelength. MoDaCor infers the representation from the Pint dimensionality of photon_units_source and converts to wavelength with uncertainty-aware BaseData arithmetic.

For measurement files with a NeXus sample-stage transformation chain, sample_z_override can also resolve the sample position from that chain:

      sample_z_override:
        type: nexus
        source: sample
        transform_path: /entry1/sample/transformations/sample_z
        component: z

component defaults to z and is extracted from the resolved lab-frame translation. The current scattering geometry calculation still models the sample position as (0, 0, sample_z); this override makes chained NeXus translations usable for the z coordinate without adding a separate preprocessing copy step.

The source value is an IoSources reference. detector_path points to the NXdetector; by default MoDaCor reads its detector_module child. Use detector_module_name when the module has a different child name.

module_origin controls how an NXdetector_module/module_offset is interpreted:

  • corner: use the resolved module offset directly.

  • first_pixel_center: treat the resolved module offset as the first pixel centre. MoDaCor converts it back to a corner origin because PixelCoordinates3D adds the half-pixel centre shift internally.

The older explicit configuration form remains supported. Use direct det_coord_*_source, pixel_pitch_*_source, and basis_* values when data are not NeXus encoded or when a pipeline intentionally overrides the file geometry.

Threshold masks#

ThresholdMask creates a uint32 mask from any BaseData entry in a selected DataBundle, not only from signal. This is useful when a correction map stored with the measurement should define invalid detector pixels. For example, to mask pixels whose flatfield correction matrix falls outside an acceptable range:

steps:
  load_flatfield:
    module: AppendProcessingData
    configuration:
      processing_key: sample
      signal_location: sample::entry/instrument/detector/flatfield
      rank_of_data: 2
      databundle_output_key: flatfield
      units_override: dimensionless
  flatfield_mask:
    module: ThresholdMask
    requires_steps: [load_flatfield]
    configuration:
      with_processing_keys: [sample]
      source_basedata_key: flatfield
      target_mask_key: flatfield_mask
      lower_bound: 0.8
      upper_bound: 1.2
      mask_mode: outside

Use mask_mode: outside to mask values below lower_bound or above upper_bound. Use mask_mode: inside to mask values within the inclusive range instead. The older threshold option is still accepted as an upper-bound alias when upper_bound is not configured. The created mask keeps the same array shape as the source_basedata_key; leading image or frame axes are not collapsed before thresholding.

The same step can create geometry masks from Q or Psi BaseData entries. For example, this masks pixels outside a radial Q range and outside an azimuthal Psi range, combines those masks, and applies the combined mask to the sample signal:

steps:
  q_range_mask:
    module: ThresholdMask
    configuration:
      with_processing_keys: [sample]
      source_basedata_key: Q
      target_mask_key: q_mask
      lower_bound: 0.05
      upper_bound: 3.0
      mask_mode: outside
  psi_range_mask:
    module: ThresholdMask
    configuration:
      with_processing_keys: [sample]
      source_basedata_key: Psi
      target_mask_key: psi_mask
      lower_bound: -0.7853981633974483
      upper_bound: 0.7853981633974483
      mask_mode: outside
  combine_geometry_masks:
    module: BitwiseOrMasks
    requires_steps: [q_range_mask, psi_range_mask]
    configuration:
      with_processing_keys: [sample]
      target_mask_key: mask
      source_mask_keys: [q_mask, psi_mask]
  apply_geometry_mask:
    module: ApplyMask
    requires_steps: [combine_geometry_masks]
    configuration:
      with_processing_keys: [sample]
      mask_key: mask
      basedata_to_mask: [signal]

Use mask_mode: inside for the inverse region of interest, for example to mask only a beamstop band or a known bad azimuthal sector while leaving the rest of the detector unmasked.

For dynamic detector data, keep the dimensionality semantics explicit:

  • Use ThresholdMask before normalization or frame averaging to create a frame-resolved raw-count mask.

  • Use ReduceMask to collapse non-detector axes while preserving NeXus bitfield reason bits. reduction: any keeps a pixel masked if it was masked in any reduced frame; reduction: all keeps only bits present in every reduced frame.

  • Use BitwiseOrMasks to combine a reduced dynamic mask with a static instrument mask before IndexedAverager.

ReduceDimensionality is intentionally signal-oriented and performs numeric mean/sum reductions. It is not a replacement for ReduceMask, because numeric averaging does not preserve integer mask bitfields.

Both modules accept axes: non_data to derive the leading reduction axes from BaseData.rank_of_data. MoDaCor treats the final rank_of_data dimensions as data dimensions, so a signal shaped (frames, singleton, y, x) with rank_of_data: 2 is reduced over axes (0, 1). Data already at its declared rank is left unchanged.

ReduceDimensionality can consume an integer mask directly without modifying the source signal. mask_key names the BaseData mask in the same DataBundle. By default every nonzero reason bit is excluded; mask_bits can restrict exclusion to one uint32 bitfield value or a list of values:

steps:
  average_frames:
    module: ReduceDimensionality
    configuration:
      with_processing_keys: [sample]
      axes: non_data
      reduction: mean
      mask_key: mask
      mask_bits: [1, 4]
      nan_policy: propagate

The mask must have an integer dtype and be broadcast-compatible with the signal. Selected elements receive zero effective weight in the signal reduction, propagated uncertainties, and scatter-derived estimators. Masked elements are omitted independently of nan_policy, so a masked NaN does not propagate but an unmasked NaN does. The source signal and mask remain unchanged.

When every contributor to an output position is masked, a mean is nan, a sum is zero, and scatter-derived estimators are nan.

Direct mask selection does not reduce the stored mask. Use ReduceMask separately if the output bundle needs a mask with the reduced dimensionality.

The optional uncertainty_estimation mapping adds scatter-derived uncertainty components under user-selected keys. Available methods are standard_deviation, standard_error_mean (mean reductions), and standard_error_sum (sum reductions). ddof defaults to 1.

steps:
  average_frames:
    module: ReduceDimensionality
    configuration:
      with_processing_keys: [sample]
      axes: non_data
      reduction: mean
      use_weights: true
      nan_policy: omit
      uncertainty_estimation:
        collision_policy: error
        estimators:
          frame_STD:
            method: standard_deviation
            ddof: 1
          frame_SEM:
            method: standard_error_mean
            ddof: 1

Estimator mapping keys such as frame_STD and frame_SEM are the exact output keys in BaseData.uncertainties. If a key already exists after normal uncertainty propagation, collision_policy may be error (the default), overwrite_existing, keep_existing, or propagate. Here, existing means the uncertainty already present at the reduced output shape. propagate combines the existing and estimated components in quadrature and therefore assumes that they are independent. An individual estimator may override the enclosing collision policy.

For a sum reduction, standard_error_sum estimates uncertainty of the sum from contributor scatter. standard_deviation remains available but describes the scatter of the contributing values rather than uncertainty of the sum:

uncertainty_estimation:
  collision_policy: error
  estimators:
    summed_repeatability:
      method: standard_error_sum
      ddof: 1

These estimates are distinct from propagation of uncertainty components already attached to the input. Known per-value uncertainties, including Poisson uncertainties, are normally best attached before reduction and allowed to propagate through the existing mean or sum formulas. The complete design and statistical contract is recorded in ReduceDimensionality uncertainty estimators.

steps:
  average_frames:
    module: ReduceDimensionality
    configuration:
      with_processing_keys: [sample]
      axes: non_data
      reduction: mean
  reduce_frame_mask:
    module: ReduceMask
    configuration:
      with_processing_keys: [sample]
      source_mask_key: threshold_mask
      target_mask_key: threshold_mask
      axes: non_data
      reduction: any

ApplyMask remains available when replacing masked signal values is preferred over direct, non-mutating selection. Its masked_value defaults to nan; explicit sentinel values remain available:

steps:
  apply_mask:
    module: ApplyMask
    configuration:
      with_processing_keys: [sample]
      mask_key: mask
      basedata_to_mask: [signal]
      masked_value: nan

When a pipeline is loaded through a runtime service using the restricted runtime policy, the module name must resolve through the service’s curated or explicit ProcessStepRegistry. Filesystem discovery of unregistered module files is disabled in that mode. If loading fails with filesystem discovery is disabled by runtime policy, export/register the step in the service registry or run a trusted local service.