Isaac Flath
Archive
Retrieval Trace Template
ConsultingAboutSubscribe

Field tool

Retrieval Trace Template

A small review kit for checking what went in, what ran, what came out, and where important records appeared next.

What it does

It makes AI search failures reviewable.

Most bad AI answers look plausible until you inspect the path. This template gives you one place to capture the handoffs between stages and write comments at each phase.

Search

Record every search call.

Capture the generated search, filters, top-k setting, scores, and returned results before judging the answer.

Context

Show what reached the answer model.

Separate found sources from the context sent to the model so you can see whether the system dropped the right source.

Notes

Write a note at the stage that matters.

Keep review simple: one editable note per stage, saved back to the trace JSON.

Screenshots

Agent and reading traces use the same review surface.

Use it for agentic investigations, docs-answer review, reranking failures, context selection, and source-grounded handoffs.

Retrieval trace viewer showing an agentic support trace graph
An agentic trace: planning, parallel searches, evidence selection, and context handoff stay visible.
Retrieval trace viewer showing a docs-reading reranking failure
A reading trace: the right policy appears in search, then reranking pushes the wrong source into context.

Workflow

Ask your agent to map your logs into it.

  1. Map your logs.

    Give your agent the schema, one example trace, and the logs for one answer.

  2. Keep the search results.

    Save what each search returned so a reviewer can see what the system found.

  3. Mark model context.

    Record the records or source text that reached the answer model, especially in multi-query agent runs.

  4. Write stage notes.

    Describe the first stage where the useful record disappeared or changed meaning.

Included

The download is a small folder you can drop into an eval workspace.

schema.json

JSON schema for one reviewable retrieval trace.

agent-mapping-prompt.md

Prompt for asking a coding agent to map product logs into a trace.

trace-review-worksheet.md

Worksheet for inspecting each stage instead of skimming the JSON.

codex-jsonl-tutorial.md

Walkthrough using the included Codex JSONL, draft trace, and cleaned trace.

scripts/codex-jsonl-to-trace.mjs

Starter adapter that turns the included Codex JSONL shape into draft trace JSON.

examples/*.json

Three focused examples for support-agent search, multi-phase hybrid search, and answer failure.

examples/chatwoot-support-corpus/

Synthetic customer-support records used by the Codex JSONL tutorial.

viewer/

Static viewer for reading the example traces in a browser.

README.md

Quickstart, field mapping, examples, and diagnosis rules.

Download

Get the retrieval trace template.

Enter your email to get the current download.