trulens.core.batch¶

trulens.core.batch ¶

Batch / offline evaluation of metrics over a pre-collected dataset.

This module provides BatchEvaluator, a way to run one or more Metrics over a tabular dataset (a pandas DataFrame or a list of dicts) without a live app or a recording session.

This is useful for:

Running evaluations in CI against a golden test set.
Benchmarking prompt or model changes across many examples.
Evaluating historical production logs offline.

Instead of extracting metric inputs from trace spans, each metric's arguments are mapped to dataset columns via Selector.from_column.

Example

import pandas as pd
from trulens.core import BatchEvaluator, Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

relevance = Metric(
    name="answer_relevance",
    implementation=provider.relevance,
    selectors={
        "prompt": Selector.from_column("query"),
        "response": Selector.from_column("answer"),
    },
)

evaluator = BatchEvaluator(metrics=[relevance])

df = pd.DataFrame({
    "query": ["What is the capital of France?"],
    "answer": ["Paris is the capital of France."],
})

results = evaluator.evaluate(df)

Classes¶

BatchEvaluator ¶

Run metrics over a pre-collected dataset without a live app.

A BatchEvaluator holds a list of Metrics whose selectors are created with Selector.from_column, mapping each metric argument to a column of the dataset.

Note

Results are returned in-memory only: nothing is persisted to the event table, so batch evaluations do not appear in the dashboard or leaderboard. For persisted, dashboard-visible evaluation of pre-collected data, use Run with mode=Mode.LOG_INGESTION instead.

PARAMETER	DESCRIPTION
`metrics`	The metrics to evaluate. Each metric's selectors must all be dataset (column) selectors created with `Selector.from_column`. TYPE: `Sequence[Metric]`
`max_workers`	Maximum number of metric evaluations to run concurrently in a thread pool. Defaults to the thread pool's own default (based on the CPU count). Set to 1 to run serially. Provider-side rate limiting (requests per minute) is handled by the provider endpoints. TYPE: `Optional[int]` DEFAULT: `None`

Functions¶

evaluate ¶

evaluate(
    data: DatasetLike,
    *,
    column_map: Optional[Mapping[str, str]] = None
) -> DataFrame

Evaluate all metrics over the dataset.

PARAMETER	DESCRIPTION
`data`	The dataset to evaluate, either a pandas DataFrame or a list of dict-like rows. TYPE: `DatasetLike`
`column_map`	Optional mapping from dataset column name to the column name referenced by the selectors, applied before evaluation. For example, `{"user_question": "query"}` makes a `user_question` column available as `query`. TYPE: `Optional[Mapping[str, str]]` DEFAULT: `None`

RETURNS	DESCRIPTION
`DataFrame`	A pandas DataFrame with one row per input row. The original columns
`DataFrame`	are preserved, and for each metric `M` three columns are added:
`DataFrame`	`M` (the score), `M_explanation` (metadata/reasons), and
`DataFrame`	`M_latency` (evaluation time in seconds). If multiple metrics
`DataFrame`	share a name, the first keeps the bare name and subsequent ones
`DataFrame`	are suffixed `_1`, `_2`, and so on.

Note

The results exist only in the returned DataFrame; nothing is written to the event table or dashboard. Use Run with mode=Mode.LOG_INGESTION to persist evaluations of pre-collected data.