Offline Batch Evaluation¶

This quickstart shows how to evaluate a pre-collected dataset with TruLens using BatchEvaluator — no live app or recording session required.

This is the right tool when you already have inputs, outputs, and (optionally) retrieved contexts sitting in a table and just want scores:

• Running evals in CI against a golden test set
• Benchmarking prompt or model changes across many examples
• Scoring historical production logs offline

Each metric argument is mapped to a dataset column with Selector.from_column(...). The same Metric objects you use with live tracing work here — you only change how their inputs are selected.

Note: BatchEvaluator returns results in the DataFrame only — nothing is written to the event table, so batch evaluations do not appear in the TruLens dashboard or leaderboard. If you want persisted, dashboard-visible results for pre-collected data, use Run with mode=Mode.LOG_INGESTION instead.

In [ ]:

# !pip install trulens trulens-providers-openai

# !pip install trulens trulens-providers-openai

In [ ]:

import os

if "OPENAI_API_KEY" not in os.environ:
    os.environ["OPENAI_API_KEY"] = "sk-proj-..."

import os if "OPENAI_API_KEY" not in os.environ: os.environ["OPENAI_API_KEY"] = "sk-proj-..."

The dataset¶

Batch evaluation works over any tabular dataset. Here we use a small pre-collected RAG dataset where each row already has a query, the retrieved contexts, and the generated answer. In practice this table might come from a CSV, a warehouse query, or an exported set of production logs.

In [ ]:

import pandas as pd

df = pd.DataFrame({
    "query": [
        "When was Yellowstone established?",
        "How deep is the Grand Canyon?",
        "What river runs through Zion Canyon?",
    ],
    "contexts": [
        [
            "Yellowstone National Park, established in 1872, was the world's first national park."
        ],
        ["The Grand Canyon is over a mile deep and up to 18 miles wide."],
        ["The Virgin River runs through Zion Canyon in southwestern Utah."],
    ],
    "answer": [
        "Yellowstone was established in 1872.",
        "The Grand Canyon is over a mile deep.",
        "The Colorado River runs through Zion Canyon.",
    ],
})
df

import pandas as pd df = pd.DataFrame({ "query": [ "When was Yellowstone established?", "How deep is the Grand Canyon?", "What river runs through Zion Canyon?", ], "contexts": [ [ "Yellowstone National Park, established in 1872, was the world's first national park." ], ["The Grand Canyon is over a mile deep and up to 18 miles wide."], ["The Virgin River runs through Zion Canyon in southwestern Utah."], ], "answer": [ "Yellowstone was established in 1872.", "The Grand Canyon is over a mile deep.", "The Colorado River runs through Zion Canyon.", ], }) df

Define metrics¶

We use the hallucination triad. The only difference from live evaluation is that each metric argument is bound to a dataset column with Selector.from_column(...).

Note that contexts is a list-valued column. collect_list=True passes the whole list to the metric in a single call; collect_list=False would call the metric once per context and aggregate.

In [ ]:

from trulens.core import BatchEvaluator
from trulens.core import Metric
from trulens.core import Selector
from trulens.providers.openai import OpenAI

provider = OpenAI(model_engine="gpt-4o-mini")

groundedness = Metric(
    name="Groundedness",
    implementation=provider.groundedness_measure_with_cot_reasons,
    selectors={
        "source": Selector.from_column("contexts", collect_list=True),
        "statement": Selector.from_column("answer"),
    },
)

answer_relevance = Metric(
    name="Answer Relevance",
    implementation=provider.relevance_with_cot_reasons,
    selectors={
        "prompt": Selector.from_column("query"),
        "response": Selector.from_column("answer"),
    },
)

context_relevance = Metric(
    name="Context Relevance",
    implementation=provider.context_relevance_with_cot_reasons,
    selectors={
        "question": Selector.from_column("query"),
        "context": Selector.from_column("contexts", collect_list=False),
    },
)

from trulens.core import BatchEvaluator from trulens.core import Metric from trulens.core import Selector from trulens.providers.openai import OpenAI provider = OpenAI(model_engine="gpt-4o-mini") groundedness = Metric( name="Groundedness", implementation=provider.groundedness_measure_with_cot_reasons, selectors={ "source": Selector.from_column("contexts", collect_list=True), "statement": Selector.from_column("answer"), }, ) answer_relevance = Metric( name="Answer Relevance", implementation=provider.relevance_with_cot_reasons, selectors={ "prompt": Selector.from_column("query"), "response": Selector.from_column("answer"), }, ) context_relevance = Metric( name="Context Relevance", implementation=provider.context_relevance_with_cot_reasons, selectors={ "question": Selector.from_column("query"), "context": Selector.from_column("contexts", collect_list=False), }, )

Run the batch evaluation¶

evaluate() returns a DataFrame with the original columns plus, for each metric, a score column, an _explanation column (the chain-of-thought reasons), and a _latency column. Metrics run in parallel across rows.

In [ ]:

evaluator = BatchEvaluator(
    metrics=[groundedness, answer_relevance, context_relevance],
    max_workers=8,
)

results = evaluator.evaluate(df)
results[["query", "Groundedness", "Answer Relevance", "Context Relevance"]]

evaluator = BatchEvaluator( metrics=[groundedness, answer_relevance, context_relevance], max_workers=8, ) results = evaluator.evaluate(df) results[["query", "Groundedness", "Answer Relevance", "Context Relevance"]]

Inputs from a list of dicts and remapping columns¶

evaluate() also accepts a list of dicts. If your column names don't match the selectors, pass column_map to rename them at eval time instead of editing your data.

In [ ]:

rows = [
    {
        "user_question": "Where is Acadia National Park?",
        "answer": "Acadia National Park is in Maine.",
    },
]

evaluator = BatchEvaluator(metrics=[answer_relevance])
evaluator.evaluate(rows, column_map={"user_question": "query"})

rows = [ { "user_question": "Where is Acadia National Park?", "answer": "Acadia National Park is in Maine.", }, ] evaluator = BatchEvaluator(metrics=[answer_relevance]) evaluator.evaluate(rows, column_map={"user_question": "query"})

Custom metrics¶

Any callable that takes text arguments and returns a float (or a (score, metadata) tuple) works. Argument names are arbitrary — the selectors decide which column feeds each argument.

In [ ]:

def answer_conciseness(answer: str) -> float:
    """Reward short answers (fewer than 20 words scores 1.0)."""
    return 1.0 if len(answer.split()) < 20 else 0.5


conciseness = Metric(
    name="Conciseness",
    implementation=answer_conciseness,
    selectors={"answer": Selector.from_column("answer")},
)

BatchEvaluator(metrics=[conciseness]).evaluate(df)[["answer", "Conciseness"]]

def answer_conciseness(answer: str) -> float: """Reward short answers (fewer than 20 words scores 1.0).""" return 1.0 if len(answer.split()) < 20 else 0.5 conciseness = Metric( name="Conciseness", implementation=answer_conciseness, selectors={"answer": Selector.from_column("answer")}, ) BatchEvaluator(metrics=[conciseness]).evaluate(df)[["answer", "Conciseness"]]

Adapting to a benchmark dataset (e.g. BEIR)¶

Any benchmark reduces to the same shape: run your retriever/generator once, collect the results into a DataFrame with query, contexts, and answer columns, then hand it to BatchEvaluator. For a BEIR corpus you would load the queries with the beir package, run your pipeline over them, and assemble the DataFrame exactly as above — the evaluation code does not change.