Evaluating Recorded Coding Agent Traces¶
Coding agents — Claude Code, Cursor, OpenCode — expose lifecycle hooks. TruLens'
client-hook runtime listens to them and writes each turn to your TruLens
database as an ordinary record: RECORD_ROOT → AGENT → GENERATION + TOOL
spans, carrying OTel gen_ai.* attributes.
This notebook assumes that has already happened. It points at a TruLens database that already contains coding-agent traces and evaluates them with TruLens' built-in agentic evaluators.
The trace is the unit that matters for coding agents. A per-answer score says almost nothing about an agent that ran eleven tools to answer one question; what you want to know is whether it picked sensible tools, whether it called them well, and whether it wasted work. So the metrics here work at the scope that question lives at: one tool call, one span event, the whole trace, or the whole session.
Companion notebook: genai_semconv_attribute_selection.ipynb
covers selector mechanics against gen_ai.* attributes in general.
Where the data came from¶
For reference only — this is the recording step, already done before this notebook starts:
pip install trulens-apps-claude # or trulens-apps-cursor, trulens-apps-opencode
export TRULENS_DESTINATION=local
export TRULENS_DATABASE_URL=sqlite:///$HOME/.trulens/client-hooks.sqlite
# Content is metadata-only by default; opt in to capture prompts and payloads.
export TRULENS_CAPTURE_CONTENT=true
export TRULENS_CAPTURE_TOOL_PAYLOADS=true
trulens-client-hooks install claude --project
Then the agent is used normally. Hook subprocesses append to a durable journal
and a detached worker drains it, so turns survive crashes and restarts.
TRULENS_DESTINATION also accepts snowflake and otlp. None of that matters
below — metrics select attributes, not storage.
Point at the database¶
# !pip install trulens trulens-providers-cortex snowflake-snowpark-python
import os
# The run lifecycle is a Snowflake-only feature; quiet it for local SQLite.
os.environ["TRULENS_MANAGE_RUNS"] = "false"
from snowflake.snowpark import Session
from trulens.core import Metric
from trulens.core import TruSession
from trulens.core.database.connector.default import DefaultDBConnector
from trulens.core.feedback.selector import Selector
from trulens.otel.semconv.trace import GenAIAttributes
from trulens.otel.semconv.trace import GenAIEvents
from trulens.otel.semconv.trace import SpanAttributes
from trulens.providers.cortex import Cortex
~/.trulens/client-hooks.sqlite is the default destination the client hooks
write to. Override CODING_AGENT_DB to point somewhere else — including a
Snowflake or Postgres URL, since nothing below depends on the backend.
One thing to be deliberate about: computing metrics writes back to this
database. Evaluation results are stored as EVAL spans next to the records
they score, which is what lets the dashboard show them together. If you would
rather not touch your live hook database, copy it first and point at the copy —
that is what the outputs below were produced against.
CODING_AGENT_DB = os.environ.get(
"CODING_AGENT_DB", os.path.expanduser("~/.trulens/client-hooks.sqlite")
)
print("reading:", CODING_AGENT_DB)
session = TruSession(
connector=DefaultDBConnector(database_url=f"sqlite:///{CODING_AGENT_DB}")
)
# Hook databases written by an older TruLens may predate the current schema.
session.migrate_database()
Choose what to evaluate¶
Coding-agent records use the client as the app name and the native client
version as the app version. get_events returns the raw spans for that scope,
which is what every metric below is computed from.
APP_NAME = os.environ.get("CODING_AGENT_APP", "claude")
APP_VERSION = os.environ.get("CODING_AGENT_VERSION", "2.1.19")
events = session.get_events(app_name=APP_NAME, app_version=APP_VERSION)
print(f"{APP_NAME} {APP_VERSION}: {len(events)} spans")
Metrics¶
From a single tool attribute¶
Tool spans carry the OTel gen_ai.tool.* attributes, so a metric can be fed
straight from one of them. Context Relevance treats each tool's result as
retrieved evidence and asks whether it was relevant to what the user asked.
Every tool span matches this selector independently, so the judge runs once per tool call and the record's score is the mean. Records whose turn used no tools have no tool spans and so no score.
snowpark_session = Session.builder.config(
"connection_name", os.environ.get("SNOWFLAKE_CONNECTION_NAME", "default")
).create()
provider = Cortex(snowpark_session=snowpark_session, model_engine="claude-sonnet-4-5")
m_tool_result_relevance = Metric(
implementation=provider.context_relevance_with_cot_reasons,
name="Tool Result Relevance",
).on({
"question": Selector.select_record_input(),
"context": Selector(
span_type=SpanAttributes.SpanType.TOOL,
span_attribute=GenAIAttributes.TOOL.CALL_RESULT,
),
})
From a span event¶
The prompt and the response are not span attributes. Per the OTel GenAI
conventions they are emitted as a gen_ai.client.inference.operation.details
span event on the generation span, gated behind content capture. Event
attributes have their own selector field, span_event_attribute:
m_answer_relevance = Metric(
implementation=provider.relevance_with_cot_reasons,
name="Answer Relevance",
).on({
"prompt": Selector(
span_type=SpanAttributes.SpanType.GENERATION,
span_event_attribute=GenAIEvents.EventAttributes.INPUT_MESSAGES,
span_event_name=GenAIEvents.CLIENT_INFERENCE_OPERATION_DETAILS,
),
"response": Selector(
span_type=SpanAttributes.SpanType.GENERATION,
span_event_attribute=GenAIEvents.EventAttributes.OUTPUT_MESSAGES,
span_event_name=GenAIEvents.CLIENT_INFERENCE_OPERATION_DETAILS,
),
})
This needs TRULENS_CAPTURE_CONTENT=true to have been set when the session was
recorded, and a TruLens recent enough to persist span events. A database
recorded without either will simply have no message events, and this metric will
score nothing.
From the whole trace¶
TruLens ships LLM judges that take the whole trace. They serialise it — with compression, so long traces stay within budget — and score it against a rubric.
- Tool Selection — were the tools chosen appropriate for the request?
- Tool Calling — were they invoked correctly, with sensible arguments?
- Execution Efficiency — did the agent reach the goal without wasted work?
Also available on any LLMProvider: plan_adherence_with_cot_reasons,
plan_quality_with_cot_reasons, tool_quality_with_cot_reasons, and
logical_consistency_with_cot_reasons.
Selector(trace_level=True) is what hands the judge the whole trace rather than
one span's attribute, and it must be the only selector on the metric.
m_tool_selection = Metric(
implementation=provider.tool_selection_with_cot_reasons,
name="Tool Selection",
).on({"trace": Selector(trace_level=True)})
m_tool_calling = Metric(
implementation=provider.tool_calling_with_cot_reasons,
name="Tool Calling",
).on({"trace": Selector(trace_level=True)})
m_efficiency = Metric(
implementation=provider.execution_efficiency_with_cot_reasons,
name="Execution Efficiency",
).on({"trace": Selector(trace_level=True)})
The hooks wrote the native session id as ai.observability.conversation_id, so
.on_conversation() sees the ordered turns of a whole coding session and scores
them together. The result attaches to the session's last turn.
m_conversation = Metric(
implementation=provider.conversation_helpfulness_with_cot_reasons,
name="Conversation Helpfulness",
).on_conversation()
Compute¶
These records were written by the hook runtime, not recorded through a TruApp,
so there is no app object holding a metric list. compute_feedbacks_on_events
evaluates a metric list against an events DataFrame — the entry point for any
already-recorded trace, whatever produced it.
session.compute_feedbacks_on_events(
events,
[
m_tool_result_relevance,
m_answer_relevance,
m_tool_selection,
m_tool_calling,
m_efficiency,
m_conversation,
],
)
session.force_flush()
records, metric_names = session.get_records_and_feedback(
app_name=APP_NAME, app_version=APP_VERSION
)
present = [name for name in metric_names if name in records.columns]
report = records[["input"] + present].copy()
report["input"] = report["input"].str[:44]
report.round(3)
The _with_cot_reasons evaluators persist the judge's reasoning alongside the
score, under <metric>_calls[...]["meta"]["explanation"]. That is where to look
when a score is surprising — the reasoning below is the metric's own account of
why it scored what it did.
# Each metric is shown on its own weakest record. Conversation-level metrics only
# attach to a conversation's last record, so they score on different rows than
# record-level ones.
for name in present:
scored = records[
records[f"{name}_calls"].apply(
lambda calls: isinstance(calls, list) and len(calls) > 0
)
].dropna(subset=[name])
if scored.empty:
continue
row = scored[name].idxmin()
calls = records[f"{name}_calls"].loc[row]
print("=" * 70)
print(name, "=", records[name].loc[row])
print("record:", records["input"].loc[row])
print("=" * 70)
print(calls[0].get("meta", {}).get("explanation"))
print()
Dashboard¶
Metrics were written back as EVAL spans next to their records, so the dashboard
shows scores alongside the trace tree — the tool calls, their arguments, and
their results.
from trulens.dashboard import run_dashboard
run_dashboard(session)