I/O API#

.. py:class:: IoSource(configuration: dict[str, ~typing.Any] = NOTHING, source_reference=’’, type_reference: str = ‘IoSource’, iosource_method_kwargs: dict[str, ~typing.Any] = NOTHING, logging_level: int = 30) :module: modacor.io.io_source

IoSource is a base class for all IO sources in the MoDaCor framework.

It provides access to a specific IO source and its associated methods.

Required configuration keys are:

   data_rank : int
       The rank of the data.
   data_key : str
       The key to access the data.
       Special note for get_static_metadata: If the key contains an @ character, the part before the @ is
       considered the group/dataset path, and the part after the @ is considered the attribute name.
   data_rank_dims : tuple[int]
       The dimensions of the data rank.
   non_data_slicing : str
       Slicing information for non-data dimensions. This must be a
       string that can be evaluated to a slice object. Multiple data
       slices can be separated by double semicolon ';;'.

  Get data from the IO source using the provided data key.

  :Parameters: * **data_key** (*str*) -- The key to access the data, e.g. '/entry1/instrument/detector00/data'.
               * **load_slice** (*Optional[ArraySlice]*) -- A slice or tuple of slices to apply to the data. If None, the entire data is returned.
                 Slicing is not yet implemented, so this will raise NotImplementedError if used.
                 Consider using the numpy.s_ or numpy.index_exp for simplifying the slicing syntax.

  :returns: *np.ndarray* -- The data array associated with the provided key. For scalars, this is a 0-d array.

.. py:method:: IoSource.get_data_attributes(data_key: str) -> dict[str, ~typing.Any] :module: modacor.io.io_source

  Get data attributes from the IO source if the format supports it else empty dict.

  :Parameters: **data_key** (*str*) -- The key to the data for which attributes are requested.

  :returns: *dict[str, Any]* -- The attributes associated with the data.
            Returns an empty dictionary if nothing available or unsupported.

.. py:method:: IoSource.get_data_dtype(data_key: str) -> ~numpy.dtype | None :module: modacor.io.io_source

  Get the data type of the data from the IO source if the format supports it else None.

  :Parameters: **data_key** (*str*) -- The key to the data for which the dtype is requested.

  :returns: *Optional[np.dtype]* -- The data type of the data associated with the provided key.
            Returns None if nothing available or unsupported.

.. py:method:: IoSource.get_data_shape(data_key: str) -> ~typing.Tuple[int, …] :module: modacor.io.io_source

  Get the shape of the data from the IO source if the format supports it else empty tuple.

  :Parameters: **data_key** (*str*) -- The key to the data for which the shape is requested.

  :returns: *Tuple[int, ...]* -- The shape of the data associated with the provided key.
            Returns an empty tuple if nothing available or unsupported.

.. py:method:: IoSource.get_static_metadata(data_key: str) -> ~typing.Any :module: modacor.io.io_source

  Get static metadata from the IO source using the provided data key.
  If the key contains an @ character, the part before the @ is
  considered the group/dataset path, and the part after the @ is considered the attribute name.
  Useful, for example, to get units from HDF5 attributes.

  :Parameters: **data_key** (*str*) -- The key to access the metadata.

  :returns: *Any* -- The static metadata associated with the provided key.

IoSources is a collection of al the defined IoSource instances to load data.

It provides the interface to register new sources and access the different data sources through a single interface.

  Get data from the specified source using the provided data key.

  The data_reference is composed of the source reference and the internal
  data reference, separated by "::".

  :Parameters: * **data_reference** (*str*) -- The reference name of the source to access.
               * **load_slice** (*Optional[ArraySlice]*) -- A slice or tuple of slices to apply to the data. If None or ellipsis, the entire data is returned.

  :returns: *Any* -- The data associated with the provided key.

.. py:method:: IoSources.get_data_attributes(data_reference: str) -> ~numpy.ndarray :module: modacor.io.io_sources

  Get data from the specified source using the provided data key.

  The data_reference is composed of the source reference and the internal
  data reference, separated by "::".

  :Parameters: * **data_reference** (*str*) -- The reference name of the source to access.
               * **index** (*int*) -- The index to access the data.

  :returns: *Any* -- The data associated with the provided key.

.. py:method:: IoSources.get_data_dtype(data_reference: str) -> ~numpy.ndarray :module: modacor.io.io_sources

  Get data from the specified source using the provided data key.

  The data_reference is composed of the source reference and the internal
  data reference, separated by "::".

  :Parameters: **data_reference** (*str*) -- The reference name of the source to access.

  :returns: *Any* -- The data associated with the provided key.

.. py:method:: IoSources.get_data_shape(data_reference: str) -> ~numpy.ndarray :module: modacor.io.io_sources

  Get data from the specified source using the provided data key.

  The data_reference is composed of the source reference and the internal
  data reference, separated by "::".

  :Parameters: **data_reference** (*str*) -- The reference name of the source to access.

  :returns: *Any* -- The data associated with the provided key.

.. py:method:: IoSources.get_source(source_reference: str) -> ~modacor.io.io_source.IoSource :module: modacor.io.io_sources

  Get the source class associated with the given name.

  :Parameters: **source_reference** (*str*) -- The reference name of the source to access.

  :returns: *IoSource* -- The source class associated with the provided name.

.. py:method:: IoSources.get_static_metadata(data_reference: str) -> ~typing.Any :module: modacor.io.io_sources

  Get static metadata from the specified source using the provided data key.

  The data_reference is composed of the source reference and the internal
  data reference, separated by "::".

  :Parameters: **data_reference** (*str*) -- The reference name of the source to access.

  :returns: *Any* -- The static metadata associated with the provided key.

.. py:method:: IoSources.register_source(source: ~modacor.io.io_source.IoSource, source_reference: str | None = None) -> None :module: modacor.io.io_sources

  Register a new source class with the given name. If no source_reference is provided, the
  source's own source_reference attribute will be used.

  :Parameters: * **source** (*IoSource*) -- The class of the source to register.
               * **source_reference** (*str*) -- The reference name of the source to register.

  Replace request-scoped source slices applied by :meth:`get_data`.

.. py:method:: IoSources.split_data_reference(data_reference: str) -> tuple[str, str] :module: modacor.io.io_sources

  Split the data reference into source reference and data key.

  The data_reference is composed of the source reference and the internal
  data reference, separated by "::".

  :Parameters: **data_reference** (*str*) -- The reference name of the source to access.

  :returns: *tuple[str, str]* -- A tuple containing the source reference and the data key.

.. py:class:: IoSink(configuration: dict[str, ~typing.Any] = NOTHING, sink_reference=’’, type_reference: str = ‘IoSink’, iosink_method_kwargs: dict[str, ~typing.Any] = NOTHING, logging_level: int = 30) :module: modacor.io.io_sink

Base class for IO sinks. Mirrors IoSource.

Sinks are registered with a resource_location (file/socket/etc.). The routed write call passes an optional ‘subpath’ after ‘::’, which may be empty.

.. py:class:: IoSinks(defined_sinks: dict[str, ~modacor.io.io_sink.IoSink] = NOTHING) :module: modacor.io.io_sinks

Registry for IoSink instances. Mirrors IoSources.

.. py:method:: IoSinks.split_target_reference(target_reference: str) -> tuple[str, str] :module: modacor.io.io_sinks

  Split 'sink_ref::subpath'. Subpath may be empty (e.g. 'export_csv::').

.. py:class:: ChunkPlan(schema_version: str, plan_id: str, total_chunks: int, expected_chunk_ids: tuple[str, …], outputs: tuple[~modacor.io.chunking.ChunkOutputLayout, …], driver: ~collections.abc.Mapping[str, ~typing.Any] = , batch_axes: tuple[int, …] = (), data_axes: tuple[int, …] = (), axis_rules: tuple[~collections.abc.Mapping[str, ~typing.Any], …] = (), bindings: tuple[~collections.abc.Mapping[str, ~typing.Any], …] = (), source_bindings: tuple[~modacor.io.chunking.ChunkSourceBinding, …] = ()) :module: modacor.io.chunking

Immutable plan-wide contract shared by every chunk.

.. py:class:: ChunkSpec(schema_version: str, plan_id: str, plan_hash: str, chunk_id: str, ordinal: int, grid_index: tuple[int, …], source_selection: tuple[~modacor.io.chunking.AxisSelector, …], expected_input_shape: tuple[int, …], placements: tuple[~modacor.io.chunking.ChunkPlacement, …]) :module: modacor.io.chunking

Immutable description of one resolved unit of chunked work.

.. py:method:: ChunkSpec.identity_dict() -> dict[str, ~typing.Any] :module: modacor.io.chunking

  Return the compact execution identity suitable for logs and traces.

Table of Contents

I/O API#