trulens.benchmark.criteria_ab_test¶

trulens.benchmark.criteria_ab_test ¶

A/B test two judge configurations against a golden set.

TruLens feedback functions accept criteria and additional_instructions to tune judge behaviour, but there is no easy way to tell whether a new criteria string actually agrees with human judgement better than the old one. CriteriaABTest runs two configurations of a feedback function over the same golden set and reports which one aligns better: side-by-side MAE / Spearman / Kendall / Brier against the ground truth, the examples where the two disagree most, a paired significance test on the score differences, and a winner.

Spearman and Kendall reuse GroundTruthAggregator so there is a single source of truth for those metrics; the paired sign-flip permutation significance test lives alongside it in groundtruth.

Example

from trulens.benchmark.criteria_ab_test import CriteriaABTest
from trulens.providers.openai import OpenAI

provider = OpenAI()
test = CriteriaABTest(
    golden_set=my_golden_set,
    variant_a={"fn": provider.relevance, "name": "default"},
    variant_b={
        "fn": provider.relevance,
        "kwargs": {"criteria": "Score strictly."},
        "name": "strict",
    },
)
report = test.run()
report.print_comparison()

Classes¶

CriteriaABTestReport ¶

Comparison of two judge configurations over the same golden set.

PARAMETER	DESCRIPTION
`name_a`	Name of variant A. TYPE: `str`
`scores_a`	Variant A scores, aligned with `scores_b`. TYPE: `List[float]`
`name_b`	Name of variant B. TYPE: `str`
`scores_b`	Variant B scores. TYPE: `List[float]`
`expected`	Optional ground-truth scores aligned with the variants. TYPE: `Optional[List[float]]` DEFAULT: `None`
`queries`	Optional per-example labels (e.g. query strings) used when listing the largest disagreements. TYPE: `Optional[List[str]]` DEFAULT: `None`

Functions¶

metrics ¶

metrics() -> Dict[str, Dict[str, float]]

Per-variant agreement with ground truth. Empty if no expected.

significance ¶

significance() -> Dict[str, float]

Mean score difference (A - B) and its permutation p-value.

top_disagreements ¶

top_disagreements(k: int = 5) -> List[Dict[str, Any]]

The k examples where the two variants differ most.

winner ¶

winner() -> Optional[str]

The variant with lower MAE vs ground truth, or None.

print_comparison ¶

print_comparison() -> None

Print the side-by-side metrics, disagreements and the winner.

CriteriaABTest ¶

Run two configurations of a feedback function and compare them.

PARAMETER	DESCRIPTION
`golden_set`	A list of dicts, each with `query`, `expected_response` and, optionally, `expected_score`. TYPE: `List[Dict[str, Any]]`
`variant_a`	A dict with a `fn` (the feedback function), a `name` and optional `kwargs` passed to `fn` on every call. TYPE: `Dict[str, Any]`
`variant_b`	A second variant in the same shape as `variant_a`. TYPE: `Dict[str, Any]`
`args_fn`	Optional function mapping a golden row to the positional arguments passed to each `fn`. Defaults to `(row["query"], row["expected_response"])`. TYPE: `Optional[Callable[[Dict[str, Any]], Tuple]]` DEFAULT: `None`

Functions¶

run ¶

run() -> CriteriaABTestReport

Score the golden set with both variants and build the report.

Rows on which either variant errors are skipped so both stay aligned.

RETURNS	DESCRIPTION
`CriteriaABTestReport`	A populated
`CriteriaABTestReport`	CriteriaABTestReport.

RAISES	DESCRIPTION
`ValueError`	If no row was scored by both variants.