dprovenance.dev / integrations / llama-index

LlamaIndex tracing
that fails the PR.

DProvenanceKit ships a drop-in callback handler for LlamaIndex: add it to the callback manager and every query or chat-engine run is recorded as a queryable, diffable trace. Then diff a candidate run against a golden baseline and gate CI when the pipeline drifts — a skipped retrieval, a changed synthesis path, a dropped sub-query. RAG regression testing, a few lines of setup.

Works with llama-index-core via the standard CallbackManager — the handler records into an active run and cleans up after it.

# 01 · install

Install

shell
$ pip install "dprovenancekit[llama-index]"

Python 3.9+ · zero third-party deps in the core · the extra pulls in llama-index-core.

# 02 · add the handler

Add the callback handler

Open a run, add DProvenanceLlamaIndexCallbackHandler to LlamaIndex's Settings.callback_manager, and run queries normally — each is recorded to the trace store.

trace_llama_index.pypython
from llama_index.core import Settings
from dprovenancekit import DProvenanceKit, SQLiteTraceStore
from dprovenancekit.integrations.llama_index import (
    DProvenanceLlamaIndexCallbackHandler,
    LlamaIndexTraceEvent,
)

kit = DProvenanceKit(LlamaIndexTraceEvent)
store = SQLiteTraceStore(LlamaIndexTraceEvent, "traces.sqlite")

with kit.run(context_id="qa-session", store=store) as run:
    handler = DProvenanceLlamaIndexCallbackHandler(run)
    Settings.callback_manager.add_handler(handler)

    response = index.as_query_engine().query("What did the author do growing up?")

store.flush()   # make the run durable before the process exits

Full reference: the LlamaIndex section of the README.

# 03 · what gets recorded

What gets recorded

DProvenanceLlamaIndexCallbackHandler is a LlamaIndex BaseCallbackHandler, so it sees the same event stream the framework emits:

  • Each on_event_start / on_event_end — LLM, retrieve, embedding, query, synthesize, function-call — becomes a typed event in execution order.
  • LlamaIndex's own event ids nest the events into a span tree.
  • The active step becomes the event's engine (llama_index), with model / tool detail in attributes.
  • Lifecycle provenance edges are emitted: DERIVED_FROM (start → end).

Two runs of the same pipeline share a fingerprint, so the fingerprint / diff / gate tooling flags a skipped retrieval, a reordered step, or a changed synthesis path.

# 04 · gate in ci

Then gate it in CI

Once runs are recorded, diff any run against a golden baseline and fail the PR when the reasoning drifts. The server-less gate CLI exits non-zero on regression, and the DProvenanceKit Regression Gate GitHub Action drops it into any workflow — see the full gate story on the main page.