Architecture Upgrade Plan#

This document tracks the design issues identified during the September 2026 architecture review. Keep the status and notes current as each item is fixed.

Status values:

  • Open: not started.

  • In progress: actively being changed.

  • Done: implemented and verified.

  • Deferred: intentionally postponed with a reason.

Items#

  1. Done Runtime API trust boundary

    The runtime API currently accepts local filesystem paths for pipeline YAML and custom source/sink class import paths from client payloads. This is acceptable only for a trusted local helper. A network-facing service needs path allowlists, disabled or registry-backed custom imports, and an explicit auth model.

    Outcome: RuntimePolicy now centralizes API trust-boundary settings. The default trusted policy preserves local-helper behavior, while the restricted policy disables pipeline.yaml_path, disables process-step filesystem discovery outside the curated/explicit registry, and disables arbitrary custom source/sink imports through kwargs.class_path. Optional read/write roots constrain file-backed source/sink paths and write_hdf output paths; restricted mode requires those roots for file-backed IO. Lightweight limits are available for session count, pipeline YAML payload size, and buffer upload size. Hard CPU/memory isolation and authentication remain deployment/container responsibilities.

  2. Done Partial rerun invalidation contract

    Dirty-step selection should be based on an explicit per-step dependency contract instead of server-owned config heuristics. Steps should declare which source refs they read and which ProcessingData keys they read or write.

    Outcome: ProcessStep.dependency_contract() now provides the partial-rerun dependency contract. Boundary steps declare exact contracts for source loading, source registration, sink registration, and sink exports. Runtime dirty-step detection now consumes the contract and expands matching seeds to downstream descendants.

  3. Done Pipeline scheduler ownership

    Pipeline currently inherits from graphlib.TopologicalSorter, which mixes graph data with one-shot scheduler state. Prefer keeping Pipeline as the graph/spec holder and creating a fresh scheduler per execution.

    Outcome: Pipeline no longer subclasses TopologicalSorter. It owns the graph and exposes create_scheduler() for fresh per-use schedulers. Existing manual scheduling helpers remain as compatibility wrappers, and the shared runner now uses a local scheduler per job.

  4. Done Process-step configuration schema

    Step configuration is split between CONFIG_KEYS and ProcessStepDescriber.arguments. Constructor-provided configuration is overwritten during initialization, and documented argument types are not centrally validated.

    Outcome: ProcessStep now builds a unified effective configuration schema from shared CONFIG_KEYS and module-specific ProcessStepDescriber.arguments. Constructor-provided configuration, pipeline YAML/spec configuration, and manual config updates all preserve user values and validate unknown keys plus top-level value types through the same path. YAML sequences are normalized to Python tuples for configuration keys that explicitly declare tuple as their runtime type. CONFIG_KEYS remains the compatibility location for shared/base options.

  5. Done Explicit in-place step execution

    calculate() is documented as returning a mapping that execute() merges, but several modules mutate shared ProcessingData before that merge. Partial rerun rollback and trace semantics would be clearer with an explicit commit boundary.

    Outcome: MoDaCor now treats in-place ProcessingData mutation as the authoritative process-step contract. execute() no longer merges returned mappings back into ProcessingData; it stores them only as optional produced_outputs bookkeeping. Built-in steps that previously relied on the merge path now update self.processing_data directly. Runtime rollback stays at session scope through partial snapshots and full-rerun fallback, avoiding per-step full data copies in the normal execution path.

  6. Done Core data metadata contracts

    DataBundle is currently a permissive dict, and BaseData arithmetic inherits axes/rank/weights from the left operand. Mixed-axis operations need an explicit metadata policy.

    Outcome: DataBundle now accepts only non-empty string keys mapped to BaseData values while preserving existing construction styles such as DataBundle(signal=bd). BaseData binary arithmetic now uses a cheap structural metadata policy: Pint still handles unit compatibility and unit algebra, while MoDaCor rejects rank and axes conflicts that can be checked without scanning full axis arrays. Metadata-neutral scalar factors and correction maps keep the primary data metadata intact, and simultaneous array-valued weights retain the existing left-operand inheritance behavior.

Update Log#

  • 2026-09-01: Created tracker and started item 2.

  • 2026-09-01: Completed item 1 with restricted runtime policy, locked registry support, IO path roots, custom IO import controls, and lightweight API limits.

  • 2026-09-01: Completed item 2. Verified with targeted .venv-dev pytest runs.

  • 2026-09-01: Started item 3.

  • 2026-09-01: Completed item 3. Verified with full .venv-dev pytest run.

  • 2026-09-01: Started item 4.

  • 2026-09-01: Completed item 4. Verified with full .venv-dev pytest run.

  • 2026-09-01: Fixed item 4 YAML tuple compatibility regression.

  • 2026-09-01: Completed item 5. Verified with full .venv-dev pytest run.

  • 2026-09-01: Completed item 6. Added cheap BaseData metadata compatibility checks and DataBundle entry validation.