trulens.benchmark.golden_set_generator

trulens.benchmark.golden_set_generator

Golden set generation from production TruLens records.

The hardest part of aligning LLM judges is building a good golden set. TruLens already stores production records — inputs, outputs and judge scores — in its database, so instead of hand-writing lists of dicts, users can sample real traffic for human annotation.

GoldenSetGenerator queries a TruSession for records (optionally filtered by app name, date range or feedback score range), samples them with a configurable strategy (uniform random, stratified across low/medium/high score buckets, or uncertainty — the records whose judge scores sit closest to the decision boundary), and exports the sample in the GroundTruthAgreement golden-set format or to CSV/JSON for external annotation tools. Once annotated, the golden set can be validated, loaded back, and persisted via TruSession.add_ground_truth_to_dataset.

Example

from trulens.benchmark.golden_set_generator import GoldenSetGenerator
from trulens.core.session import TruSession

session = TruSession()
generator = GoldenSetGenerator(session, seed=42)

# Sample 50 records, stratified by existing relevance scores.
sample = generator.sample(
    n=50,
    app_name="my_rag_app",
    strategy="stratified",
    feedback_name="relevance",
)

# Export for annotation.
sample.to_csv("golden_set_to_annotate.csv")

# After human annotation, validate, load back and persist.
annotated = generator.load_annotations("golden_set_annotated.csv")
generator.save_golden_set("my_golden_set", annotated)

Classes

GoldenSetSample

A sample of records ready for human annotation.

Rows carry the golden-set columns (query, expected_response and an empty expected_score to be filled by an annotator) plus provenance columns (record_id, app_name, app_version, ts, and the sampled judge's name and score when a feedback function was used for sampling).

PARAMETER	DESCRIPTION
df	Sample rows, one per record. TYPE: DataFrame

Functions

to_df

to_df(include_provenance: bool = True) -> DataFrame

Return the sample as a DataFrame.

PARAMETER	DESCRIPTION
include_provenance	Keep the provenance columns (record id, app name/version, timestamp, judge score). When False, only the golden-set columns are returned. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
DataFrame	A copy of the sample rows.

to_list

to_list(include_provenance: bool = False) -> List[Dict]

Return the sample as a list of golden-set dicts.

The result matches the GroundTruthAgreement golden-set format: [{"query": ..., "expected_response": ..., "expected_score": None}].

to_csv

to_csv(path: str, **kwargs) -> None

Write the sample to a CSV file for external annotation.

PARAMETER	DESCRIPTION
path	Destination file path. TYPE: str
**kwargs	Extra arguments passed to pandas.DataFrame.to_csv. DEFAULT: {}

to_json

to_json(path: str) -> None

Write the sample to a JSON file (list of row objects).

GoldenSetGenerator

Samples production records from a TruSession for golden-set curation.

PARAMETER	DESCRIPTION
session	The TruSession whose database holds the records. TYPE: TruSession
seed	Optional seed making sample reproducible. TYPE: Optional[int] DEFAULT: None

Functions

sample

sample(
    n: int,
    app_name: Optional[str] = None,
    app_version: Optional[str] = None,
    strategy: str = STRATEGY_RANDOM,
    feedback_name: Optional[str] = None,
    start_time: Optional[Any] = None,
    end_time: Optional[Any] = None,
    min_score: Optional[float] = None,
    max_score: Optional[float] = None,
    decision_boundary: float = 0.5,
    max_records: Optional[int] = None,
) -> GoldenSetSample

Sample records for annotation.

PARAMETER	DESCRIPTION
n	Number of records to sample. If fewer records match the filters, all matching records are returned with a warning. TYPE: int
app_name	Only sample records from this app. TYPE: Optional[str] DEFAULT: None
app_version	Only sample records from this app version. TYPE: Optional[str] DEFAULT: None
strategy	One of "random" (uniform), "stratified" (equal samples from the low/medium/high score buckets) or "uncertainty" (records whose scores are closest to decision_boundary). TYPE: str DEFAULT: STRATEGY_RANDOM
feedback_name	Feedback function whose scores drive the "stratified"/"uncertainty" strategies and the min_score/max_score filters. Required for those; ignored otherwise. TYPE: Optional[str] DEFAULT: None
start_time	Only sample records at or after this timestamp (anything pandas.Timestamp accepts). TYPE: Optional[Any] DEFAULT: None
end_time	Only sample records at or before this timestamp. TYPE: Optional[Any] DEFAULT: None
min_score	Only sample records with feedback_name score >= this. TYPE: Optional[float] DEFAULT: None
max_score	Only sample records with feedback_name score <= this. TYPE: Optional[float] DEFAULT: None
decision_boundary	Center of the "uncertainty" strategy; records are ranked by abs(score - decision_boundary). TYPE: float DEFAULT: 0.5
max_records	Cap on how many records to fetch from the database before sampling. TYPE: Optional[int] DEFAULT: None

RETURNS	DESCRIPTION
GoldenSetSample	A GoldenSetSample
GoldenSetSample	with expected_score left empty for annotation.

RAISES	DESCRIPTION
ValueError	On an unknown strategy, a missing/unknown feedback_name where one is required, or n < 1.

load_annotations

load_annotations(
    source: Union[str, DataFrame, Sequence[Dict]],
    require_scores: bool = True,
    score_range: Optional[Tuple[float, float]] = (0.0, 1.0),
) -> DataFrame

Load and validate an annotated golden set.

PARAMETER	DESCRIPTION
source	Path to an annotated CSV or JSON file (as produced by GoldenSetSample.to_csv / GoldenSetSample.to_json), or an equivalent DataFrame / list of dicts. TYPE: Union[str, DataFrame, Sequence[Dict]]
require_scores	Require every row to carry a numeric expected_score. TYPE: bool DEFAULT: True
score_range	Inclusive (low, high) bounds scores must fall in; pass None to skip the range check. TYPE: Optional[Tuple[float, float]] DEFAULT: (0.0, 1.0)

RETURNS	DESCRIPTION
DataFrame	A DataFrame ready for
DataFrame	save_golden_set,
DataFrame	with expected_score (and record_id when present) also folded
DataFrame	into a per-row meta dict so they survive persistence.

RAISES	DESCRIPTION
ValueError	On missing columns, empty queries, missing scores (when required) or out-of-range scores.

save_golden_set

save_golden_set(
    dataset_name: str,
    annotated: Union[str, DataFrame, Sequence[Dict]],
    dataset_metadata: Optional[Dict[str, Any]] = None,
) -> int

Persist an annotated golden set as a TruLens dataset.

PARAMETER	DESCRIPTION
dataset_name	Name of the dataset to create or extend. TYPE: str
annotated	Annotated golden set — anything load_annotations accepts, or the DataFrame it returned. TYPE: Union[str, DataFrame, Sequence[Dict]]
dataset_metadata	Optional metadata stored with the dataset. TYPE: Optional[Dict[str, Any]] DEFAULT: None

RETURNS	DESCRIPTION
int	The number of ground truth rows written.