0001 — Pydantic as the parse target, or only as the schema

Status Proposed Date 2026-09-11

Context

pyDEXPI models the DEXPI schema as generated Pydantic classes (495 under DEXPI 2.0 when this was decided, 502 under 2.0.1), and those same classes are also the target the loaders construct into. Two independent pressures now push against the second of those roles.

Tolerant loading fights strict construction

A parser accepts a document as it is and reports what was wrong. Pydantic refuses to construct an invalid object. Every leniency feature therefore has to be implemented by evading the component doing the work, and the evasion has accumulated:

tree_ref_base/loader.py	lines
at 17156296, before the process work	733
at 62504113	733
after the process-loading work	1087

Roughly 250 of those 1087 exist to work around validation rather than to parse DEXPI:

• ~139 lines — _compose_without_unusable_values, _unusable_value_locations and _missing_attributes_only: catch a ValidationError, determine which value it blames, drop it, retry, and give up after a bounded number of passes.
• ~107 lines — _extract_composition_chain: a topological sort over two kinds of dependency that deliberately cuts a required-reference edge and defers it to a later pass.
• model_construct, which bypasses validation outright, and _deferred_required_refs, referenced at eight sites.

Two facts make this structural rather than incidental.

DEXPI 2.0 Process mandates a reference cycle. ProcessConnection.source / .target and Port.connectorReference are required references to each other, so no construction order exists in which either object can be built complete. Pydantic requires a complete object at construction; the format guarantees one cannot be had. This is not a defect to be fixed upstream.

Real files omit mandatory attributes. 410 of the 523 objects in the CO2 terminal example whose class requires Identifier omit it. Rejecting them, with the descendants they take along, leaves 146 of the 959 objects a lenient load builds from the file (47 of its 109 process steps, no port and no stream; the file has 987, and 28 sit under names the model does not have), so strict=False exists — implemented by catching the validator and reconstructing.

The reviewer who knows this code best reported that he can no longer follow it manually without significant thought.

Profiles fight static classes

Separately, an open question is how pyDEXPI should respond to DEXPI 2.0 profiles, which extend the metamodel at runtime with new attributes on existing classes and new classes. One option is to load DEXPI into a loose graph and validate outside — the same proposal as the loading one, reached from the other direction.

Both pressures push on one decision, which is why they are recorded together.

What Pydantic is unambiguously right for

Any option must preserve these, because they are the library's value rather than incidental:

• The generated classes are the DEXPI schema in executable form — the core promise.
• model_fields introspection is load-bearing in many modules: the toolkits, the exporters, the diff, label and quantity handling.
• validate_assignment=True makes an invalid in-place change fail loudly, which is exactly right for the toolkits and for any code that changes a model.
• Shipping type information to users argues for leaning harder on the classes, not less.

The question is not whether to keep Pydantic. It is whether the same object should also be the parse target.

Options considered

A. Keep Pydantic at the loading boundary, accept the complexity

What happens today. The loader constructs generated classes directly and evades validation where a real document requires it.

For: no migration; one representation; the strict path stays genuinely strict.

Against: every new tolerance lands as another layer inside the constructor path, and the hardest part of the loader is the part that exists to evade the validator. Profiles have no answer at all under this option.

B. Split collection from validation behind a permissive intermediate

Parse into a tolerant representation, then cast into the schema as an explicit, separate step.

For: the cycle problem disappears. With collection separated from construction, every object is read unvalidated, references are wired, and the model is constructed once with everything known — removing the two-kind topological sort and the edge cutting rather than tidying them. Leniency becomes a property of one boundary instead of a behaviour threaded through the constructor. Profiles get a natural home, since the permissive form need not know the schema.

Against: a real inversion, not a refactor. graph_loader already has dexpi_to_graph and graph_to_dexpi, so the round trip exists — but the graph is derived from a built model, and using it as the parse intermediate means reversing that to XML -> graph -> model. It adds a second representation for contributors to hold in mind, and the public API would have to say which one callers see.

C. Abandon the static classes

Represent DEXPI as a graph throughout, with validation entirely external.

For: matches where the DEXPI initiative appears to be heading; profiles become trivial.

Against: discards the library's main promise. No static types, so shipping type information becomes impossible, and every consumer of model_fields is rewritten. Not proportionate to the problem.

Decision

Option A for now — keep Pydantic as the parse target — recorded deliberately rather than defaulted into, and with explicit triggers for revisiting.

The complexity is real and measured, but it is currently contained: it lives in one module, behind a documented contract, pinned by tests that state what is current behaviour rather than correct behaviour. Option B is the right end state if the pressure continues, and it should not be started as a refactor between other work — it changes what the loaders produce, and the public API question has to be answered first.

Consequences

• tree_ref_base/loader.py stays the most intricate module in the repository. Changes to it warrant more review than their diff size suggests.
• New leniency should extend the existing helpers rather than adding another layer. A change that cannot be expressed through _missing_attributes_only or _compose_without_unusable_values is evidence for revisiting this decision, not a reason to invent a third mechanism.
• Profiles remain unanswered, and are not settled by this record.
• The cycle cutting in _extract_composition_chain is load-bearing and must keep its insertion ordering: a plain set there makes the loaded model depend on hash ordering between runs.

Revisit when

Any one of these is sufficient:

1. A third pressure arrives — a requirement that also wants a permissive representation. Profiles landing in real files would be it.
2. The loader crosses a complexity threshold. As a concrete marker: another 50% growth in loader.py, or a fourth distinct validation-evasion mechanism.
3. The public API question is settled. Option B is much cheaper to attempt once the boundary callers depend on is written down.
4. A correctness bug is traced to the interleaving of collection and construction — for instance, a document that loads differently depending on which reference edge was cut.