Regression testing for AI agents: catch reasoning drift before it ships

# 01 · the failure mode

Output tests pass. The reasoning regressed.

The nastiest agent bugs don't throw. You reword a system prompt, bump a model version, or reorder a tool list, and the agent keeps producing plausible answers. Nothing crashes. No assertion fires. But somewhere inside the run it stopped calling its verification step, or started looping its search tool three times where one call used to do.

Traditional testing can't see this, because asserting on final output means matching a string or schema an LLM produced — flaky by construction, and blind to how the answer was assembled. An answer produced without the verify step can look identical to one that was checked. The regression isn't in the output; it's in the execution path. That is the gap LLM regression testing exists to close: treat the reasoning itself — the sequence of steps, tools, and engines — as the thing under test.

# 02 · the definition

What is prompt regression testing?

The framing works because structure is far more stable than text. Two healthy runs of the same agent rarely produce identical tokens, but they take a recognizable path: plan, search, rank, verify, decide. When that path changes without you intending it to, something in your prompt, model, or orchestration changed the agent's behavior — and that is exactly what a regression test should flag. It is not a replacement for evals (more on that in the FAQ); it's the structural complement that can run deterministically on every pull request.

# 03 · golden runs

The golden-run workflow

The mechanics mirror snapshot testing, applied to reasoning instead of rendered output:

• Record a run you trust as a trace — every step, in order, with its span structure.
• Save that trace as the golden baseline (a plain SQLite file you can commit).
• Re-run the agent after every prompt edit, model bump, or refactor.
• Diff the candidate against the golden and fail when the structure drifts.

Here is the whole loop with a single import — this is the README's quick start, unedited:

from dprovenancekit import trace

# 1. Record an execution
with trace("Agent Workflow"):
    with trace("Retrieve Documents"):
        # your retrieval code here
        pass
    with trace("Verify Claims"):
        # your verification code here
        pass

# 2. Save the trace
trace.save("golden_run.sqlite")

# 3. Print a structural explanation
trace.explain()
# --- Execution Trace (b4f8d2…) ---
# ▶ Started Agent Workflow
#   ▶ Started Retrieve Documents
#   ✔ Finished Retrieve Documents
#   ▶ Started Verify Claims
#   ✔ Finished Verify Claims
# ✔ Finished Agent Workflow

# 4. Catch regressions when the logic changes: rerun the (now buggy) workflow,
#    then diff the current run against the saved golden baseline
trace.diff("golden_run.sqlite")
# --- Trace Diff (Golden vs Current) ---
# ❌ Missing step: Verify Claims

The same record → explain → diff loop, with more context, is on the main page. Under the hood it powers anomaly detection, CI gating, and visual trace analysis.

# 04 · fingerprints

Run fingerprints: the cheap signal

Before you reach for a full step-by-step diff, there's a one-string-compare shortcut: the run fingerprint, a structural identity of the execution path. DProvenanceKit computes it as a SHA-1 over each event's type-and-engine signature in sequence order — so equal fingerprints mean the same typed steps ran through the same engines in the same order.

Reordered tools, a skipped retrieval step, a dropped verify step: all of them change the fingerprint. That makes it a cheap regression signal you can check constantly — equal fingerprints and you're done; different fingerprints and you run the full alignment to see exactly which steps were added, removed, or reordered, and how severe that is. Because the hash covers structure rather than token output, two runs that phrase their answers differently but take the same path fingerprint identically.

# 05 · a real catch

Worked example: the research agent that stopped verifying

A research agent, recorded twice into one local SQLite file. The golden run takes the path plan → search.web → rank → verify.claims → decide. The candidate — after a PR that "just" reworded the planner prompt — loops search.web three times, then skips verify.claims entirely. The server-less gate CLI selects the newest run per context id and diffs them; no backend, no account:

$ dprovenancekit gate --db traces.sqlite \
    --golden-context golden --candidate-context candidate

Regression gate: FAIL
  severity: high (strength 0.95); max allowed: none
  fingerprint: differs (5e616565a219… vs b6e1023b183b…)
  per-step changes:
    added: search.web, search.web
    removed: verify.claims
  engine: Critical reasoning steps removed: verify.claims

$ echo $?
1

The non-zero exit code is the point: this command drops into any pipeline as a hard gate — the CI gate guide covers GitHub Actions and GitLab wiring end to end. And for invariants that should hold on every run, not just against a baseline, two anomaly rules ship out of the box — Tool Drop (a required step never ran) and Looping (a step repeated past a threshold) — configured through a JSON rule registry:

{
  "rules": [
    { "type": "tool_drop", "required_step": "safety_check" },
    { "type": "looping", "step": "web_search", "max_repeats": 5 }
  ]
}

$ dprovenancekit anomalies --db traces.sqlite --rules rules.json — runs the rules locally or on every PR, no golden required.

# 06 · in the test suite

LLM regression testing in pytest

The same gate is one assertion inside any test. Give it a golden run and a candidate run; it aligns them and fails with a readable diagnostic when the candidate diverged:

from dprovenancekit.testing import RegressionGate, assert_no_regression

# Strict by default: any removed, added, or changed (ambiguous) step fails,
# and a removed CRITICAL step is a HIGH-severity regression.
assert_no_regression(golden=golden_run, candidate=candidate_run)

# Or build a reusable policy and inspect the verdict without raising:
report = RegressionGate(allow_divergent_steps=True).check(golden_run, candidate_run)
assert report.passed, report.reasoning

Regressions carry a severity — none / low / medium / high — and a removed or reordered CRITICAL step is a HIGH-severity regression (reordering a critical step can invert a dependency). Loosen the gate with max_regression_level (gate on severity only) or allow_divergent_steps (tolerate benign per-step changes), or pass a custom evaluator to define what "equivalent" means — for example, ignore volatile payload fields like token counts. One subtlety worth knowing: detecting reordered steps requires a span-aware profile such as AlignmentProfile.developer_debug_v1; the default linear profile treats a pure reorder as still-matching, so pass the span-aware profile when order is part of what you're guarding.

The bundled pytest plugin removes the remaining plumbing. One fixture records the block as a candidate run and pins the baseline for you:

def test_research_agent(golden_trace):
    with golden_trace("research-agent"):
        run_my_agent()   # anything using @traced / record_event / an adapter

$ pytest --dprov-update-golden   # record (or intentionally update) the baseline, commit it
$ pytest                         # every later run gates against tests/goldens/research-agent.sqlite

Baselines are SQLite files committed next to your tests (directory configurable via the dprov_golden_dir ini option). Gate options can be set per test — e.g. golden_trace("name", max_regression_level="high").

# 07 · your stack

Works with your stack — or with no framework at all

Everything above operates on traces, so the only integration question is how runs get recorded. For a hand-written agent loop, the core instrumentation ships with zero dependencies:

from dprovenancekit import InMemoryTraceStore, traced, traced_run, record_event

@traced
def search(query): ...

@traced
def answer(question, sources): ...

store = InMemoryTraceStore()
with traced_run(store, context_id="ticket-42"):
    sources = search(question)
    record_event("plan.chosen", {"strategy": "rag"})
    reply = answer(question, sources)

@traced records a <name>.start / .end / .error event pair per call in its own span, and works on plain functions, async def, generators, and async generators. Instrumentation never changes behavior — exceptions pass through unchanged, and outside a traced_run the decorator is transparent. Already on a framework? Adapters record the same trace shape:

• LangChain / LangGraph — pip install "dprovenancekit[langchain]", then attach DProvenanceTracer / DProvenanceCallbackHandler.
• OpenAI Agents SDK — pip install "dprovenancekit[openai-agents]"; officially listed in the OpenAI Agents SDK docs. See the integration page.
• LlamaIndex — pip install "dprovenancekit[llama-index]".
• CrewAI — pip install "dprovenancekit[crewai]".
• Plus integration modules for Google GenAI, FastAPI, Jupyter, and MCP.

# 09 · get started

Get started

$ pip install dprovenancekit

• v0.4.0 on PyPI · Python 3.9+ · zero third-party dependencies in the core · Apache-2.0.
• Validated: 284 tests; precision / recall / F1 of 1.000 across 8 standard + 5 adversarial regression-detection scenarios (dprovenancekit evaluate), matching the original Swift implementation case-for-case.
• CI-ready: a GitHub Action on the Marketplace and a GitLab CI template in the repo.

Next: wire it into CI → View on GitHub

The SDK, CLI, and Action are free and open source. A hosted trace visualizer (span tree, payload inspector, side-by-side semantic diff) is in early access at app.dprovenance.dev.