Common regression patterns, and how DProvenanceKit detects each

# 01 · the two detection layers

Two layers: what changed, and whether it's a regression

Every pattern below is caught by one or both of two engines that run over the same pair of recorded runs. TraceDiffEngine answers what changed — which step signatures were added or removed. TraceAlignmentEngine answers is it a regression — it maps each critical step in the baseline to an equivalent counterpart in the candidate, in causal order, and only fails when that mapping breaks. The diff is the cheap, human-readable co-signal; the alignment is the verdict a gate acts on.

// Layer 1 — structural diff: WHAT changed (added / removed step signatures).
let diff = TraceDiffEngine<FoundationModelTraceEvent>()
    .diff(base: base, comparison: candidate)   // floored at .structural
for change in diff.changes {
    print(change.kind, change.typeIdentifier)      // e.g. removed  fm_tool_call
}

// Layer 2 — semantic alignment: IS it a regression.
let engine = TraceAlignmentEngine(configuration: FoundationModelAlignment.configuration())
let alignment = engine.align(base: base, comparison: candidate)
print(alignment.regressionRisk.level, alignment.regressionRisk.reasoning)

• The diff runs a collection difference over typeIdentifier::engineName signatures — linear in the number of events, and diff.isIdentical is a one-line pre-check before you pay for alignment.
• Alignment scores every base × comparison pair, assigns matches highest-score-first, then derives the risk verdict from the equivalence outcome and critical-only order — not from display labels.
• FoundationModelAlignment.configuration() defaults to the span-aware developerDebugV1 profile (semantic threshold 0.75); the strict-audit profile strictAuditV1 is linear with threshold 0.99.

# 02 · pattern 1

Dropped tool call

A critical step in the baseline has no counterpart at all in the candidate. The alignment interpreter files it as .removed; the risk pass collects its type into removedCriticalTypes and the verdict is HIGH. This is the canonical silent failure: nothing crashes, no error is thrown, the reply is fluent — the agent just stopped calling a tool and answered from the model's prior.

The bundled FoundationModelsRegressionDemo is exactly this. A weather agent calls getWeather before an OS/model update and skips it after:

Structural diff (baseline → candidate):
  removed: tool call · getWeather
  removed: tool output · getWeather

Semantic alignment:
  regression risk: HIGH — Critical reasoning steps removed: fm_tool_call

CI gate: FAILED — reasoning regression detected

1. The diff is the co-signal: getWeather's call and output are both removed — struck through in red in the Explorer and the WebVisualizer export.
2. The alignment is the verdict: the tool call is a .critical step (fm_tool_call), so its removal drives RegressionRisk.high.
3. The reasoning string names only the critical removals: fm_tool_call, not fm_tool_output — the tool output is .structural, so it shows in the diff but does not itself trigger the gate.

# 03 · pattern 2

Reordered steps

Two critical steps whose causal order inverts — x precedes y in the baseline but follows it in the candidate — are collected into reorderedCriticalTypes, and the verdict is HIGH with the reasoning "Critical reasoning steps reordered: …". An invoice generated before its customer is created is not equivalent to the same events in the right order, even though the exact same steps ran.

# 04 · pattern 3

Path divergence — same input, different route

The same request is answered by a different reasoning path: some steps appear, others disappear. The run's structural fingerprint differs, so the diff shows both added and removed steps. This is cheap to spot before any deep alignment, and precise to assert with the query DSL.

• Cheap: TraceDiffEngine.diff() is linear over the typeIdentifier::engineName signature set; diff.isIdentical tells you two runs took the same shape without materializing an alignment. Each persisted run also carries a stored structural fingerprint on its runs row.
• Precise: assert the route you expect with requiring(step:), missing(step:), matching(step:where:), and the ordering operators requiring(step:followedBy:) / requiring(step:precededBy:).

// Find runs that concluded a conflict without ever evaluating the document.
let runs = try await store.queryRuns(
    TraceQueryDSL<MyAIDecision>()
        .requiring(step: "conflictDetected")
        .missing(step: "documentEvaluated")
)

# 05 · pattern 4

Silent answer-only drift

The output text changes but nothing else does — no crash, no error, the same steps in the same order. This is the hardest case, because a string-equality test either flakes or is simply absent. DProvenanceKit catches it two ways.

1. A critical step changed beyond equivalence. The step still binds to a same-type counterpart, but the equivalence model scores that pair below the profile's semanticThreshold and the payloads differ. Its type lands in changedCriticalTypes and the verdict is HIGH: "Critical reasoning steps changed beyond equivalence: …".
2. Caused by a skipped tool. When the drift is a downstream symptom of a dropped step, Pattern 1 fires alongside it — the removed critical step is the dominant, higher-strength signal.

let alignment = engine.align(base: base, comparison: candidate)
let risk = alignment.regressionRisk
print(risk.level, risk.reasoning)
// high  Critical reasoning steps changed beyond equivalence: decisionReached

# 06 · pattern 5 · bonus

Structural invariant violations

Some regressions need no baseline at all: a single run is wrong on its face. A conflict reported with no supporting document evaluation, an authorization concluded without the check that should justify it — "a conclusion without its supporting step". MissingSupportRule plus AnomalyDetector flag these rule-first, over any run in the store.

let rule = MissingSupportRule<MyAIDecision>(
    name: "UnsupportedConflict",
    whenPresent: "conflictDetected",
    isMissing: "documentEvaluated"
)
let anomalies = try await AnomalyDetector(store: store)
    .detectAnomalies(rules: [rule])

• MissingSupportRule compiles to the query requiring(step:).missing(step:) — it is the batteries-included form of Pattern 3's precise assertion, packaged as a reusable rule.
• The step names are your event typeIdentifiers, so a rule survives payload refactors; conform your own type to AnomalyRule for any other shape.
• The same rule can be registered live against a LiveTraceQueryEngine to fire the moment a run starts matching, instead of in a batch sweep.

# 07 · reading the verdict

Reading a verdict, and how it maps to an exit code

Every alignment produces one RegressionRisk: a level, a strength (a normalized 0–1 heuristic, not a probability), and a human-readable reasoning string that names the offending step types. The three failure shapes each carry their own reasoning prefix — removed, reordered, or changed beyond equivalence — so the string alone tells you which pattern fired.

let risk = alignment.regressionRisk
print(risk.level, risk.reasoning)
// high  Critical reasoning steps removed: fm_tool_call

// The bundled demo's --gate flag fails the build on medium-or-worse;
// in practice that means .high, since .none is the only other emitted level.
if risk.level == .high || risk.level == .medium {
    FileHandle.standardError.write(Data("reasoning regression: \(risk.reasoning)\n".utf8))
    exit(1)   // non-zero exit breaks CI the same way a failing test would
}

That non-zero exit is the whole point: a dropped, reordered, or drifted reasoning step fails the pull request instead of shipping green. The Swift-native CI gate guide covers the recipes and the honest boundary of what dpk does and doesn't gate.

# 08 · coverage & honesty

Coverage, and what falls outside

These five are the curated failure modes the engine is built to catch — the shapes in the conformance corpus, where perfect scores are evidence the engine behaves correctly on the regressions it is designed for. That is a controlled diagnostic corpus, not statistical generalization to arbitrary traces. Being precise about the boundary is the point.

• Trace-observable behavior only. Equivalence is decided over the recorded causal graph. Real-world effects and uninstrumented internal state that never entered an event are out of scope by construction.
• No model-internal deliberation. "Reasoning" here means the instrumented path — prompts, tool calls, outputs, decisions, your own events. The model's hidden internal thinking is not visible and is not claimed.
• Not truthfulness. The verdict tells you the reasoning structure and observable content changed, or a step vanished — not that an answer is factually correct.
• Not completeness. The engine sees the steps you instrumented. A regression in a step you never recorded is a blind spot, not a pass.
• Relative to a chosen observation model. Each AlignmentProfile defines its own equivalence relation via a deterministic scoring procedure — reproducible, not a claim of metaphysical semantic correctness.