A CI gate for reasoning regressions

# 01 · the missing pipeline stage

Why a gate, not a dashboard

A tracing dashboard tells you what the agent did after it ran — which, for a regression, means after the pull request merged and shipped. By then you are bisecting commits while users see the broken behavior. A dashboard is a rear-view mirror; nothing in it stops the next bad merge.

Deterministic code solved this with a pipeline stage decades ago: tests run on every pull request, and a failure blocks the merge. Reasoning deserves the same stage. The only new artifact is the trace — instead of asserting on a return value, you assert on the execution path: which steps ran, in what order, through which tools. Record a known-good run once, diff every candidate against it, and exit non-zero when a critical step vanished, moved, or materially changed.

The full loop is five stages, and most stacks are missing the last two:

• record — capture the run as a diffable trace.
• diff — TraceDiffEngine reports which steps appeared or disappeared.
• detect — TraceAlignmentEngine rates the change none / low / medium / high.
• gate — a non-zero exit on medium/high blocks the merge.

# 02 · the python guide, and where it stops

Relationship to the existing site guide

The published CI gate for LLM agents guide is the Python story, and it is complete for a Python stack. If your agent runs in Python, use that guide — nothing below replaces it. It covers:

• The server-less dprovenancekit gate CLI — point it at a SQLite database, name a golden and a candidate context id, and it exits non-zero on regression.
• The GitHub Action that wraps the CLI and posts a sticky PR comment with the per-step diff.
• A GitLab CI template with parity to the Action.
• Golden-run hygiene through the pytest plugin (--dprov-update-golden).

That whole path is database-based golden/candidate gating: record both runs into SQLite, then let one CLI command diff them. This page adds the Swift-native path the guide does not cover — because, as the next section is blunt about, the Swift CLI is a different tool with a narrower job.

# 03 · what dpk does and does not gate

The honest Swift-side gap

If you come from the Python guide expecting a gate --db command on the Swift side, stop here first. There isn't one, and pretending otherwise would send you gating the wrong thing.

So the Swift toolkit gives you three distinct gate-shaped jobs, and it helps to keep them straight:

• Gate your agent — the shipped FoundationModelsRegressionDemo --gate (Recipe 1), or a ~20-line gate target you own (Recipe 2). This is the one that catches your reasoning drift.
• Gate the engine — dpk evaluate --gate (Recipe 3). Proves DProvenanceKit's own detector still scores as expected on its corpus; think of it as a dependency-pin check, not an agent check.
• Verify a shipped trace — dpk verify (Recipe 4). Proves an attested trace is intact and signer-pinned, offline. A different guarantee from drift detection.

# 04 · recipe 1 — the shipped demo gate

Fastest: run the demo gate DProvenanceKit already ships

The package ships a runnable end-to-end scenario: a weather agent that silently stops calling its getWeather tool after an OS/model update and answers from the model's prior instead of live data. The reply is fluent and plausible; a unit test stays green; the reasoning is wrong. In gate mode the demo exits non-zero when it catches it, so it drops straight into a CI step.

$ swift run FoundationModelsRegressionDemo          # prints the analysis, exits 0
$ swift run FoundationModelsRegressionDemo --gate   # CI mode: exits non-zero on a medium/high regression

The tail of what it prints — the alignment verdict and the gate decision the exit code follows:

Semantic alignment:
  regression risk: HIGH — Critical reasoning steps removed: fm_tool_call

CI gate: ❌ FAILED — reasoning regression detected

A dropped tool call is a .critical step, so removing it raises the regression risk to high — and --gate turns that into exit(1). The demo also writes fm-regression.json, a web-export document you can drop into the Explorer to read the diff visually.

Use this recipe as-is to prove the wiring in a fresh CI job before you point the same mechanism at your own agent (Recipe 2).

# 05 · recipe 2 — a gate target for your agent

Your agent: a ~20-line gate executable

The demo constructs its two runs by hand. For your own agent you record a golden run and a candidate run — into two SQLite files, or two context ids in one — then diff and align them with the exact same engine calls. Add an executable target that loads both runs and exits on a high/medium verdict:

import Foundation
import DProvenanceKit

// Two runs you recorded earlier: the committed golden baseline, and the
// candidate this PR's code produced. `AgentStep` is your own TraceableEvent.
let golden = try SQLiteTraceStore<AgentStep>(fileURL: goldenURL)
let candidate = try SQLiteTraceStore<AgentStep>(fileURL: candidateURL)

guard let base = try await golden.queryRuns(TraceQueryDSL()).first,
      let cand = try await candidate.queryRuns(TraceQueryDSL()).first
else { fatalError("no runs recorded") }

// A reordered CRITICAL step is caught in every alignment mode — the linear
// strictAuditV1 profile included; .developerDebugV1 is a fine default here.
let config = AlignmentConfiguration(
    profile: .developerDebugV1,
    equivalenceEvaluator: AnyEquivalenceEvaluator<AgentStep>(
        identifier: "agent-eq-v1",
        evaluator: { $0 == $1 ? 1.0 : ($0.typeIdentifier == $1.typeIdentifier ? 0.8 : 0.0) }
    )
)

let diff = TraceDiffEngine<AgentStep>().diff(base: base, comparison: cand)
let alignment = TraceAlignmentEngine(configuration: config).align(base: base, comparison: cand)

if alignment.regressionRisk.level == .high || alignment.regressionRisk.level == .medium {
    FileHandle.standardError.write(Data("reasoning regression: \(alignment.regressionRisk.reasoning)\n".utf8))
    exit(1)
}

regressionRisk.level is a RegressionRisk.Level with cases none / low / medium / high; reasoning is the human-readable cause the engine attached. The diff result is there for a per-step log line; the gate decision itself rides on the alignment verdict. In practice the shipped engine emits only .none or .high today — the .medium arm above is future-proofing, not a state you will observe yet (see common regression patterns).

# 06 · recipe 3 — gate engine conformance

Pin the detector: dpk evaluate --gate

This recipe gates a different artifact: the detector, not your agent. dpk evaluate --gate runs DProvenanceKit's standard and adversarial alignment corpus and fails the build if any case regresses; with --min-f1 it also fails when F1 drops below a baseline you pin, on both datasets. Run it when you bump the DProvenanceKit dependency, so an engine change can't silently weaken the detector every other gate relies on.

$ swift run dpk evaluate --gate --min-f1=1.0   # fail if the built-in corpus regresses or F1 < 1.0

Argument parsing is fail-closed: anything unknown, malformed, duplicated, or out of range exits 2 before any work runs — and --min-f1 without --gate is itself an error. A typo therefore cannot quietly turn the check off while still reporting success.

# 07 · recipe 4 — verify a shipped trace

Attestation in CI: dpk verify

If your pipeline ships a trace as an artifact — a signed record of how a decision was reached — you can prove it is intact and came from a signer you pinned, offline, as a job step. This is integrity, not drift detection: it answers "is this the trace that was signed, by the key I trust?", not "did the reasoning change?".

$ swift run dpk verify --in=decision.attestation.json --trusted-key=<trusted-key-id>

• --in is required — the attestation document (or, with --proof-pack, a proof pack) to verify.
• --trusted-key pins the signer: a 64-hex-character key id, repeatable. Omit it and verification proves integrity only — the signature is valid but the signer identity is not pinned.
• The document comes from your app via TraceAttestationDocument.signed(run:using:); swift run dpk attest-demo writes a sample you can practice against, and prints the signer key id to pin.

# 08 · the github actions workflow

Wrap the gate on pull_request

The whole thing is one job on macOS runners — swift run the gate, and upload the web-export so a reviewer can open the diff in the Explorer without digging through logs. This wraps Recipe 1; swap the run: line for your own gate target from Recipe 2.

name: agent-regression
on: pull_request

jobs:
  gate:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v4

      # Recipe 1: the shipped demo gate. Swap in your own gate target (Recipe 2).
      - name: Reasoning-regression gate
        run: swift run FoundationModelsRegressionDemo --gate

      # Attach the web-export so reviewers can open the diff in the Explorer.
      - name: Upload trace diff
        if: always()            # upload even when the gate fails — that's when you want it
        uses: actions/upload-artifact@v4
        with:
          name: reasoning-diff
          path: fm-regression.json

macOS runners because the package builds against Apple platforms. if: always() keeps the artifact even on a failing gate, which is exactly the run you want to inspect. For a Recipe 2 target, point path: at whatever web-export your gate writes.

# 09 · golden-run hygiene

Treat the golden like a snapshot, because it is

A golden trace is a snapshot of your agent's reasoning, and the workflow is the one you already know from snapshot testing. The failure mode to avoid is re-recording the baseline to silence a failure you can't explain — that is the regression, waved through.

• Commit the golden, or archive it. A recorded run is a small, deterministic SQLite file — check tests/goldens/agent.sqlite into the repo, or (if it is large or environment-specific) publish it as a CI artifact the gate downloads. Either way the gate has a fixed baseline to diff against.
• Re-baseline only on intentional change. When the agent is supposed to reason differently, re-record the golden with the same recording step your candidate uses, and review the change in the PR — because the gate itself can diff the two goldens, a baseline update is reviewable, not opaque.
• Keep the candidate ephemeral. Record it fresh in the job from the PR's code; only the golden is version-controlled.
• Trust the detector first. Before you gate on it, run Recipe 3 — dpk evaluate --gate --min-f1=1.0 — so you know the engine still scores 1.000 on its own corpus on your machine, not just in a README.
• Mixed Swift/Python codebase? Trace Specification v1 keeps the two implementations behaviorally equivalent, so a golden recorded against one stays meaningful across both.

← Back to the docs hub Common regression patterns Add to an existing agent The Python guide Open the Explorer