Skip to content

Explanation: Parsing and ETL Structure in the Parser App

Concept

The parser app implements a simplified ETL pipeline – Extract, Transform, and Load – with a strong focus on the parsing step. At its core, the app allows users to upload files from laboratory experiments or field measurements, apply custom parsers to interpret the content, and then transform the parsed results into structured database entities that can be stored reliably for later analysis.

The idea behind this architecture is the separation of concerns. Instead of treating file upload, parsing, transformation, and database loading as one monolithic process, the app breaks them into clear stages. Each stage has its own responsibility:

  • Extract: Handling files as raw input.
  • Transform
    • Parsing: Turning raw input into structured domain objects.
    • Mapping: Converting those domain objects into database-ready models.
  • Load: Persisting the final models into the database.

This design provides clarity, maintainability, and flexibility. It allows the app to support many different test formats and research workflows without constantly rewriting the underlying logic.

How it works

The workflow of the app can be understood in four main steps, which follow the ETL principle.

1. File upload (Extract)

Researchers begin by uploading files generated by instruments or test rigs. These files might be in different formats – CSVs with tensile test data, JSON from sensor systems, XML from automated machines, or even proprietary log files. The app does not assume any specific structure at this point. Its only job is to receive and store the raw data, leaving interpretation for the next stage.

2. Parsing step (Transform – structure discovery)

Parsing is the stage where meaning is applied to raw data. The app allows users to provide custom parsers. A parser is a piece of logic that knows how to interpret a particular file format and produce structured domain objects.

For example, consider a CSV file containing test results from material samples. A MaterialSampleParser would read each row of the file and create Material domain objects with attributes such as sampleId, composition, and testDate. Similarly, a TestResultParser could take a laboratory log file and output TestResult objects with fields like testId, strengthValue, and temperature.

By allowing user-defined parsers, the app avoids being tied to one specific format. Instead, it becomes a flexible framework where parsing is pluggable.

3. Mapping step (Transform – semantic mapping)

Once the parser has produced domain objects, the app transforms these into database entities. This involves mapping the structure of the domain objects to the database schema. Data types are normalized, units are converted, relations between samples and tests are established, and constraints (like uniqueness or non-null fields) are applied.

For instance, the Material domain object created by the parser might be transformed into a Material database object that matches the schema of the database table Material. The transformation ensures consistency between user-defined logic (parsing) and the technical requirements of the storage system.

4. Loading step (Load)

Finally, the database entities are persisted into the target system. At this stage, the app handles all the technical details of saving objects: inserting records, maintaining referential integrity, and reporting any errors that occur during the load process.

The loading step is intelligent about whether to create new objects or update existing ones. If a parser sets the code attribute to be the one of an object instance, the system recognizes this as a reference to an existing object and updates its properties rather than creating a duplicate. This allows parsers to handle both initial data import and subsequent updates with the same workflow.

By the end of this step, the raw test file uploaded by the researcher has been fully integrated into the database in a safe and consistent way.

Why this structure?

The app could, in theory, skip some of these steps. One might ask: why not simply load raw files directly into the database? The answer lies in the advantages of separating parsing, transformation, and loading.

  1. Flexibility By introducing custom parsers, the app can handle a wide variety of test formats without modifying its core. A new instrument or file format only requires writing a new parser, while the rest of the system remains unchanged.

  2. Reusability Parsers are reusable components. A parser written for one type of test file can be used repeatedly, regardless of how the transformation or loading steps evolve. This modularity prevents duplication of logic.

  3. Robustness Parsing errors are detected early, before they affect the database. If a file cannot be interpreted, the process fails gracefully at the parsing stage instead of creating corrupt or inconsistent database entries.

  4. Maintainability Each stage is easier to reason about and maintain when responsibilities are clearly separated. Developers working on parsers do not need to worry about database internals, and vice versa.

  5. Performance By isolating parsing and transformation, the system can optimize specific parts of the pipeline. For example, parsing can be parallelized across large sets of test data, while loading can be tuned for batch inserts.

Design considerations

Designing the parsing and ETL pipeline involves several important decisions.

  • Parser interface All parsers follow a consistent contract: they accept files as input and return a collection of domain objects such as material samples or test results. This consistency allows the app to integrate any parser seamlessly.

  • Error handling Validation is critical at the parsing stage. Files may contain missing values, invalid formats, or corrupted rows. Detecting these issues early prevents them from propagating into the database and ensures data integrity for downstream analysis.

  • Extensibility The modular design makes it easy to support new instruments and data sources. Adding support for a new file format requires only a new parser, not a rewrite of the transformation or loading steps.

  • Trade-offs Modularity adds complexity. Developers must define and manage parsers carefully. There is also a performance cost when data passes through multiple transformations. However, the long-term benefits of reliability and flexibility outweigh these costs.