Skip to content

0002 — Comparing model states, and reviewing them in Git

Status Accepted (2026-09-25) Date 2026-09-25

Context

Anyone keeping DEXPI models under version control asks different questions about change:

  1. What differs between these two model states? Revision comparison, review UIs, APIs.
  2. What should a reviewer see in GitHub or GitLab? A stable text view that diffs well.
  3. Is this model exactly the one I stored? A fingerprint that can be persisted.

The first question had a partial answer and the second none. A separate tool, dexpidiff, prototypes the second.

What existed

pydexpi.diff compares two models. PyDexpiDiffEngine matches objects through a pluggable SemanticEquality (id_equality, TagEquality), then reports added, removed, moved and modified objects as DiffResult. When this was decided the results were stdlib dataclasses; AttributeChange carried the live objects of both models, so a result could not be serialised. Composition-list order was ignored and reference lists were compared as sets.

dexpidiff renders a Proteus file as INI-like text (categories such as EQUIPMENT and PIPING, tag-named sections, reversible _XX name escaping) and diffs two such files section by section. It is built on pyDEXPI 1.x and model_dump(), and carries no licence.

Evidence that shapes the design

Snapshots are not history. A property changed 8 → 10 → 12 → 8 gives an empty diff between the first and last state, and three changes in between. Both answers are correct; they are answers to different questions, and a comparison of states answers only the first.

Stable ids exist in only one of the two domains. Loading the same file twice re-mints the id of every object the file did not identify. Measured by loading each of the 35 TrainingTestCases twice:

objects new id on reload
all 2803 1996
with a make_id-shaped id 1996 1996
with a tag (display_name.get_tag) 286 0
non-graphics 808 155

So in XML-derived models the unstable ids are exactly the generated-looking ones, the EngineeringModel root among them, and every tagged object keeps the id its file gave it. Models maintained as JSON, by contrast, keep every id — including make_id-shaped ids of objects created in code, which look generated but are stable once saved. The shape of an id alone therefore cannot say whether to trust it; the caller must.

The comparison was blind to things that change what a model says. List order (z-order; the flow order of a segment's items and connections), a move between two attributes of the same parent, and — in the generated data-type equality — the order of a list inside a data type, such as the languages of a MultiLanguageString. (A list-valued data attribute such as PolyLine.points is compared element by element, so its order is seen.)

The existing document hash is not stable across processes. hashing.canonical_json_hash uses Python's salted hash(), and its canonical serialisation sorted data-type lists by hash() as well. It cannot be persisted.

Options considered

One change format for everything. Make the Git text a diff format, or read it back into a model change. Rejected: the Git text is lossy by design, and a comparison of two loaded models is the exact answer.

Import dexpidiff's code. Rejected: no licence, a pyDEXPI 1.x data model, and an identity scheme (tagName → subTagName → list index) that collides on duplicate tags and renumbers every untagged sibling on insertion. Its concepts are reused.

For the Git projection: nested INI, YAML, canonical JSON, or a flat line format. Nested INI and YAML encode the hierarchy in section paths or indentation, so a rename or reparenting rewrites the whole subtree. Canonical JSON diffs as brace and comma noise and gives poor hunk headers. A flat, one-block-per-object text keeps a change inside the changed object's block.

Decision

Separate representations, with one-way bridges between them:

State comparison     -> DiffResult               pydexpi.diff
Exact state          -> model_digest             pydexpi.hashing
Git readability      -> text projection          pydexpi.versioning

model A, model B --id_diff--> DiffResult
model           --projection--> text --git diff--> review          (never back to a model)
  • A. DiffResult becomes a versioned Pydantic model. format_version and a matching label (id, tag, custom — never a callable's name) travel with it; values stay live in memory for existing callers and serialise to a documented wire form (pydexpi.loaders.value_codec). No timestamps or other nondeterministic fields.
  • B. hashing.model_digest fingerprints the exact serialized state, list order included, as a SHA-256 digest that is the same in every process and on every machine, so it can be stored.
  • C. The Git projection lives in a new pydexpi.versioning package, which depends on the diff's identity rules and on nothing that depends on it.
  • D. Identity is domain-specific. Acting on a diff needs stable ids (id_diff); a tag-matched diff is for display. The projection takes an id_policy: trust every id (models kept as JSON) or only ids not shaped like make_id ids (third-party XML; a heuristic on the id's shape, borne out by the measurement above, not knowledge of provenance), falling back to deterministic structural identities.
  • E. The projection is a flat, line-based text, one block per object, versioned, not round-trippable. The canonical DEXPI file stays the source of truth; the projection is a review artifact.

Whether the projection is committed next to the model, generated by a Git textconv driver, or produced in CI is a repository-workflow choice pyDEXPI does not make.

Addendum: the projection's identity for untrusted objects (measured)

Decision D left the key for objects without a trusted id to a measurement. On the TrainingTestCases (changed projection lines; 35 reloads, 3 property changes on untrusted objects, 11 insertions into the largest untrusted sibling group, 1 rename of a tank created in code):

key reload property insertion rename
anchored: nearest trusted ancestor + path + own content + tag 0 18 83 10
parent token + content ("structure first") 0 4032 83 8
tag first, then content 0 4088 83 8
position among siblings 0 6 685 4

Anchored keys were adopted. Keying on the parent's token cascades a change into every descendant; keying on position renumbers later siblings; keying a lone child by position alone renames it when a second sibling appears. The price of anchored keys is local: changing or renaming an untrusted object changes that object's own token (the tag is part of its key), while a trusted object's token is its id alone.

Consequences

  • There are two equalities, and the tests name which one they use: semantic equality (id_diff reports no changes) and exact serialized-state equality (hashing.model_digest, which is order-sensitive and stable across processes).
  • Comparing two independently exported third-party files produces a diff whose ids cannot be trusted to name the same object twice.
  • The projection's identity for objects without a trusted id is a trade-off with no perfect answer; the choice is made by measuring churn and recorded in docs/mkdocs/versioning.md.
  • dexpidiff is not integrated (closed as won't-do). It has no licence that would allow incorporating it, and pyDEXPI provides loading, the semantic diff and the Git projection itself, as library functions and in the pydexpi CLI. Its concepts remain prior art; there is no second diff engine to consolidate.

Revisit when

  1. Tag-matched comparisons need stable answers, for merging third-party revisions. That needs an explicit answer to which id a matched pair keeps.