Versioning: diffs, digests and Git¶

pyDEXPI has separate ways of talking about change. They answer different questions, and mixing them up is the most common source of wrong answers — so this page starts with the distinction, and the reasoning behind it is recorded in ADR 0002.

Question	Representation	Where
What differs between two model states?	Semantic diff — `DiffResult`	`pydexpi.diff`
Is this exactly the model I stored?	Digest — `model_digest`	`pydexpi.hashing`
What should a reviewer see in Git?	Git projection — stable text	`pydexpi.versioning`

Semantic diff¶

A vs B  ->  what differs?

A diff compares two model states. It knows nothing about how one became the other. It reports objects added, removed, moved to another parent, and modified, with the attribute changes of each. Objects are matched between the two models by a semantic equality: id_diff matches on stable ids, and PyDexpiDiffEngine(TagEquality()) matches on engineering tags.

from pydexpi.diff import id_diff

diff = id_diff(old_model, new_model)
for modified in diff.modified_objects:
    print(modified.tag, [change.attribute_name for change in modified.changes])

As a machine-readable artifact¶

A DiffResult is a versioned Pydantic model, so it crosses process and API boundaries as plain JSON and reads back:

from pydexpi.diff import DiffResult

payload = diff.model_dump(mode="json")
same = DiffResult.model_validate(payload)

The payload carries its own metadata and nothing nondeterministic — no timestamps:

format_version — "2"; a reader should refuse a version it does not know. Format 2 added AttributeChange.ordered -- an ordered reference list is written in order, not as sorted ids -- and compares the lists DEXPI declares ordered whether or not the diff tracks order. A stored format-1 payload still reads, for display, but it does not say what a format-2 diff of the same models says: recompute it before acting on it.
matching — "id", "tag" or "custom": how objects were paired. Only an "id" diff pairs each object with the one carrying the same id.
track_order — whether reorders of a segment's flow order were compared (below).

In memory an AttributeChange holds the live values of both models; serialized, a reference is the target's id, a reference list the sorted ids -- or, when the change says ordered, the ids in their order -- and anything else the shared value wire form (see Values on the wire). A result read back therefore carries wire values, and serializing it again gives the same payload.

What a change concerns¶

Every entry has a domain — "conceptual", "graphics" or "metadata" — decided once, in pydexpi.diff.domains, by where the object lives: anything under diagram or shapeCatalogues is graphics, conceptualModel.metaData and the EngineeringModel's own attributes are metadata, the rest is conceptual. has_semantic_changes, has_graphic_changes and has_metadata_changes read it, so stamping the modification date — which a save may do — is not reported as an engineering change.

Order¶

Most lists are sets in all but name, and the diff compares them as such. The lists whose order means something come from two places, both read through pydexpi.toolkits.list_order:

DEXPI declares them ordered (is_ordered on the generated field): the drawing's geometry and z-order -- the groups of a RepresentationGroup or Diagram, a type group's elements, a polyline's points --, a PhysicalQuantityVector's values, and a template's ListOfMaterialComponents.component, against which a fraction vector is read position by position. id_diff always compares these by position: a reordered composition list is an AttributeChange(category="composition") with the member ids in their old and new order, a reordered reference list a "reference" change, a reordered vector a "data" change of its value. Each says ordered=True where the attribute itself is the list.
pyDEXPI orders them: a PipingNetworkSegment's items and connections (flow order). id_diff(a, b, track_order=True) also reports their reorders.

Before the ordered-list change (diff format 2) only z-order and flow order counted, and only with track_order; a swapped pair of component ids went unreported. (A swapped pair of fraction values was already reported, as a changed value of the vector's QualifiedValue.)

Part of a model¶

Two different questions, with two different answers at a subtree's boundary:

Filter a whole-model diff — id_diff(a, b, scope=[pump_id]), or diff.for_objects(ids) with ids from pydexpi.diff.subtree_ids. Everything is compared, then the result is narrowed. A valve moved out of a scoped segment is still reported as moved.
Compare two subtrees on their own — PyDexpiDiffEngine(id_equality).diff_objects(old_segment, new_segment). Only the subtrees are traversed. The same valve is reported as removed: where it went is out of view.

diff.semantic_only(), graphics_only(), metadata_only(), in_domains(...) and filter(predicate) narrow a result the same way; each returns a new DiffResult with the same metadata.

A move between two attributes of the same parent (a value moved from pressure to temperature of one process step) is a move too; MovedObject names the container attribute on both sides.

Git projection¶

model  ->  stable, human-readable text  ->  git diff

The canonical DEXPI file (XML or JSON) stays the source of truth. The Git projection is a review artifact: a deterministic text rendering of one model state, designed so that a small engineering change produces a small textual diff. It is deliberately not round-trippable.

from pydexpi.versioning import model_to_projection

text = model_to_projection(model)                     # XML-derived models (default)
text = model_to_projection(model, id_policy="all")    # JSON models with stable ids
text = model_to_projection(model, include_graphics=False)

The format is one block per object, sectioned by where objects live:

# pydexpi-projection 2

## conceptualModel.pipingNetworkSystems

PipingNetworkSegment@758eb4aff283
  connections = [Pipe@c12277c7aa94]
  sourceItem -> Nozzle@4ee07a0fc5c5
  targetItem -> Nozzle@bb77dc341f46

Pipe@c12277c7aa94
  sourceItem -> Nozzle@4ee07a0fc5c5
  targetItem -> Nozzle@bb77dc341f46

A header is Class@token, followed by the object's tag if it has one. The body lists its set attributes, sorted, indented two spaces, so Git's default hunk header names the object.
Children are listed in their parent and follow it as blocks. A list keeps its order only where the order means something (see Order above); every other list is sorted. That holds inside a value too: a vector's values keep their order.
A unit is written class-qualified, 60.0 PressureGaugeUnit.Bar: a gauge and an absolute pressure never render alike. Format 2 changed both; format 1 sorted every list inside a value and wrote the bare literal, so every quantity line differs between the two.
Format 2 also sorts an object's own unordered data lists (MaterialTemplate.phaseLabel), which format 1 wrote in file order: reordering one is no longer a changed line.
References are written name -> Class@token: connectivity is part of the review.

Tokens. A token is 12 hex digits of a SHA-256 — never Python's hash() — of the object's identity. An object with a trusted id is keyed by it, and nothing else. id_policy="all" trusts every id, which is right for models whose ids survive saving. id_policy="source" trusts every id not shaped like a make_id id: a heuristic, not provenance, which holds across the TrainingTestCases (every id re-minted on reload has that shape, every other one is stable). An object without a trusted id is anchored: keyed by its nearest trusted ancestor, the attribute path down from it, its own content and its tag, with an ordinal only among exact duplicates — so renaming an untrusted object does change its token, while renaming a trusted one never does. That choice was measured on the TrainingTestCases (changed lines in the projection):

key for an untrusted object	3 property changes	11 insertions	1 rename
anchored (the default)	18	83	10
parent token + content	4032	83	8
tag first, then content	4088	83	8
position among siblings	6	685	4

Keying on the parent cascades a change into every descendant; keying on position renumbers every later sibling on an insertion (the 83 lines are the inserted blocks themselves). Anchored keys cost only the changed object's own lines. Every tagged object in the corpus has a trusted id, which is why a rename costs the same few lines under every scheme there.

The same file loaded twice gives the same text, and two models id_diff(..., track_order=True) calls equal give the same text.

A textual Git diff is never read back. To get the exact change between two Git revisions, load both models and compare them:

Git revision A  ->  DEXPI model A  \
                                     ->  semantic diff
Git revision B  ->  DEXPI model B  /

Identity: which ids can be trusted¶

Everything above depends on recognising "the same object" in two states, and that depends on where the models came from:

Models maintained as JSON keep every object id across saves and loads, including the id_<hex> ids make_id mints for objects created in code. Stable-id comparison (id_diff) is exact here.
Models loaded from XML keep the ids the file supplies, but objects the file did not identify get a fresh id on every load. In the TrainingTestCases corpus that is 1996 of 2803 objects — every one of them recognisable by base_model_utils.is_generated_id — while all 286 tagged objects keep their ids. Two loads of the same file therefore do not compare equal under id_diff; use tag-based matching, or the projection's source-id policy.

Two kinds of equality¶

"These two models are the same" means one of two things, and code and tests should say which:

Semantic equality — id_diff(a, b).has_changes is False. List order is ignored where neither DEXPI nor pyDEXPI gives it meaning (Order), and so is pyDEXPI's flow order unless track_order=True; reference lists are compared as sets unless they are ordered.
Exact serialized state — pydexpi.hashing.model_digest(a) == model_digest(b). A SHA-256 digest of the serialized model with every list in its order. It is the same in every process, so it can be stored, for example as the version a later change was made against.

canonical_json_hash is neither: it ignores the order of unordered lists, keeps that of ordered ones, and it is Python's salted hash(), valid only within one process. canonical_json_equality compares the same canonical form; it is what MCP compare_models answers with.

Values on the wire¶

Diffs write attribute values in one shared JSON form, pydexpi.loaders.value_codec: primitives as themselves, enum members as their value, datetime as ISO 8601, objects as {"dexpi_class", "data"} envelopes holding their subtree, and references to objects as {"dexpi_ref", "dexpi_class"} records naming them without copying them.

A number that is not finite is written as the JSON string "NaN", "Infinity" or "-Infinity", wherever it sits, an envelope included. A DEXPI Double is an xsd:double, so a loaded model may hold one, and JSON has no number for it: pydantic wrote null, which reads back as another value. Every value in a diff is therefore strict JSON, however it is serialized afterwards. The string is read back by the attribute it is assigned to, as an enum's value and a timestamp are: a float field reads "NaN" as the number, a str field as the text, and no generated class has a field that takes both. Two NaNs are equal on the wire, so a diff does not take a NaN for a change. A JSON document spells such a number the same way, so documents and diffs read alike; hashing.model_digest follows neither spelling and is the same as before both.