Pipeline Operations Backlog#

This page tracks runtime-service feature upgrades and the structural refactors that keep the pipeline-operations code maintainable as the service grows.

Status legend:

  • planned

  • in_progress

  • done

Current focus#

The U6 to U8 maintenance window is now closed.

  • U6 is complete.

  • U8 is now complete.

  • U10 is now complete.

  • U9 and U7 remain planned feature upgrades.

  • The immediate follow-up is documentation tightening and the next operational persistence and streaming features.

Upgrade ledger#

  1. U1 modacor session CLI wrapper for session lifecycle and processing calls Status: done Notes: implemented in src/modacor/cli.py as modacor session ... with commands: list, create, delete, status, set-source, delete-source, process, reset, runs. Benefit: high Complexity: medium Reason: removes raw HTTP friction and improves day-to-day operator ergonomics.

  2. U2 Source patch convenience endpoint (POST /sessions/{id}/sources/patch) Status: done Notes: implemented in src/modacor/server/api.py with OpenAPI + docs updates. Benefit: medium Complexity: low Reason: simplifies common single-source update workflow.

  3. U3 Source templates/profiles (e.g. MOUSE, SAXSess) Status: done Notes: implemented as built-in profiles with GET /v1/source-templates and session-level source_profile validation. Benefit: high Complexity: medium Reason: prevents misconfiguration and standardizes source expectations per instrument family.

  4. U4 Dry-run endpoint for invalidation preview Status: done Benefit: very high Complexity: low-medium Reason: immediate visibility into dirty-step decisions; improves trust and debugging without executing a run.

  5. U5 Richer run summaries (dirty/skipped steps, fallback reason, timings) Status: done Notes: run metadata now includes dirty_steps, skipped_steps, step_durations_s, elapsed_s, and fallback fields. Benefit: high Complexity: medium Reason: improves observability and post-mortem diagnostics.

  6. U6 “Last sample” shortcut endpoint Status: done Benefit: high Complexity: low Reason: maps the most frequent operation to a single focused API call. Notes: implemented as POST /v1/sessions/{id}/sample plus CLI modacor session set-sample ....

  7. U8 Health/readiness split endpoints with runtime metrics Status: done Benefit: high Complexity: low Reason: improves deployability and operational safety (orchestration probes, monitoring). Notes: implemented as GET /v1/health for liveness and GET /v1/readiness for service usability plus runtime metrics (session_count, active_run_count, error_session_count, error_session_ids, last_updated_utc).

  8. U10 Latest error diagnostics endpoint Status: done Benefit: medium-high Complexity: low-medium Reason: speeds triage and recovery after failed runs. Notes: implemented as GET /v1/sessions/{id}/errors/latest, returning the current session error state plus the latest recorded failed-run diagnostics. The CLI wrapper now exposes the same payload via modacor session last-error --session-id ....

  9. U9 Persistent session store Status: planned Benefit: high Complexity: medium-high Reason: preserves runtime definitions/state across restarts; requires careful state/version handling.

  10. U7 Improved event streaming (persistent WS + SSE option) Status: planned Benefit: medium-high Complexity: high Reason: valuable for real-time UIs and remote control loops but introduces more protocol/runtime complexity.

Structural maintenance window#

These tasks are intentionally scheduled between U6 and U8 so the runtime service grows on a cleaner foundation.

  1. S1 Runtime-service module split Status: done Scope: src/modacor/server/api.py now stays focused on FastAPI route bindings and HTTP translation, while src/modacor/server/runtime_service.py owns session orchestration, src/modacor/server/planning.py owns dry-run and dirty step planning, src/modacor/server/io_utils.py adapts session source registrations, and src/modacor/server/errors.py defines framework-agnostic service errors. Benefit: high Complexity: medium Reason: separates transport concerns from runtime behavior and makes service logic easier to test in isolation.

  2. S2 Python 3.12 floor alignment Status: done Scope: package metadata, CI, tox, and installation documentation all target Python 3.12+. Benefit: high Complexity: low Reason: removes unsupported-version ambiguity and matches the codebase’s typing/runtime requirements.

  3. S3 Test-package separation Status: done Scope: package-internal tests were moved out of src/modacor/tests into the top-level tests/ tree, and distribution discovery now excludes modacor.tests*. Benefit: high Complexity: medium Reason: keeps shipped packages lean and avoids coupling runtime imports to test helpers.

  4. S4 Process-step export and docs unification Status: done Scope: modacor.modules.__all__ now exports every supported ProcessStep, and the module-doc generation tests assert that exported steps and generated docs stay in sync. Benefit: high Complexity: medium Reason: removes the split between runtime discovery and reference documentation.

  5. S5 Shared runtime I/O helpers for CLI and service execution Status: done Scope: src/modacor/io/runtime_support.py now centralizes source builders, sink builders, and shared HDF export handling used by both modacor run and the runtime service. Benefit: high Complexity: medium Reason: eliminates duplicated source/HDF handling and makes future runtime upgrades less error-prone.

Notes#

  • Upgrades are implemented incrementally, with structural maintenance work allowed between feature upgrades when it reduces future implementation risk.

  • This file is the source of truth for progress tracking.

  • Companion design notes live in docs/pipeline_operations/runtime_service_api.md.