Pipeline graphs

A pipeline is a directed acyclic graph whose nodes are ProcessStep instances and whose edges express execution prerequisites. YAML uses requires_steps to declare those edges:

name: minimal_demo
steps:
  uncertainty:
    module: PoissonUncertainties
    configuration:
      with_processing_keys: [sample]
  normalize:
    module: DivideDatabundles
    requires_steps: [uncertainty]
    configuration:
      with_processing_keys: [sample, exposure]

Pipeline.from_yaml(...) validates the structure and resolves module names through a ProcessStepRegistry. Step identifiers are normalized to strings and must be unique. Every declared dependency must identify another step.

Normal execution uses run_pipeline_job(...), which creates a fresh topological scheduler for each run. Independent ready nodes may be returned together, but the standard runner executes them deterministically in its scheduler loop. For interactive debugging, pipeline.create_scheduler() exposes the underlying prepare, get_ready, done, and is_active protocol.

The graph captures ordering, while each step’s dependency contract captures which sources and processing paths it reads and writes. Both are needed for a correct partial rerun.

Pipelines can render Mermaid, DOT, or native editable draw.io representations. Use a short short_title to add scientific purpose to a node without changing its identifier or module.

For pipelines authored with step_blocks, both renderers group generated nodes automatically. The block is shown as an outer cluster or subgraph, each for_each item is shown as a processing lane, and the nodes and edges remain the real expanded execution graph. DOT also aligns matching local steps across lanes as stages where Graphviz permits it.

dot_source = pipeline.to_dot()
mermaid_source = pipeline.to_mermaid()

Native draw.io export uses Graphviz for the initial layout and emits uncompressed mxGraphModel XML. Nodes, dependency connectors, blocks, and item lanes remain individually editable. Graphviz’s dot executable must be installed:

from pathlib import Path

drawio_xml = pipeline.to_drawio(direction="TB")
Path("pipeline.drawio").write_text(drawio_xml, encoding="utf-8")

The optional dot_executable argument selects a non-default Graphviz binary, and pixels_per_inch controls conversion from Graphviz points to draw.io canvas units. Set group_step_blocks=False for a flat editable graph.

TD is accepted as a top-down direction alias and emitted as Mermaid’s canonical TB spelling so the direction also remains valid inside nested item subgraphs.

Grouping is a visual interpretation of each node’s origin metadata. It does not create a block node or change scheduling and dependency semantics. To inspect the traditional flat graph, disable it explicitly:

flat_dot = pipeline.to_dot(group_step_blocks=False)
flat_mermaid = pipeline.to_mermaid(group_step_blocks=False)
flat_drawio = pipeline.to_drawio(group_step_blocks=False)

Mermaid uses private renderer-local identifiers such as node_0; the visible labels still contain the complete expanded step ids. This prevents distinct pipeline ids containing punctuation from colliding after Mermaid identifier sanitization.