trulens.benchmark.score_distribution¶

trulens.benchmark.score_distribution ¶

Score distribution diagnostics for feedback functions.

A feedback function can post a reasonable mean error yet still be useless if it returns nearly the same score for every input. A judge that scores everything 0.4-0.6 cannot rank examples no matter how good its average looks.

ScoreDistributionAnalyzer runs a feedback function over a golden set and reports how well the resulting scores discriminate: the score histogram, a calibration curve against the expected scores, discrimination metrics (standard deviation, unique score count, normalized entropy) and the predicted spread per expected-score bucket. It also flags common judge pathologies such as poor discrimination, leniency bias and binary (bimodal) scoring. It complements the scalar metrics in GroundTruthAggregator (mae, brier, ece) by diagnosing the shape of a judge's output rather than a single error number.

Example

from trulens.benchmark.score_distribution import ScoreDistributionAnalyzer
from trulens.providers.openai import OpenAI

analyzer = ScoreDistributionAnalyzer(
    feedback_fn=OpenAI().relevance,
    golden_set=my_golden_set,
)
report = analyzer.run()
report.print_summary()
report.plot()  # matplotlib histogram + calibration curve

Classes¶

ScoreDistributionReport ¶

Diagnostics computed from predicted (and optional expected) scores.

PARAMETER	DESCRIPTION
`predicted`	Scores produced by the feedback function, each in `[0, 1]`. TYPE: `Sequence[float]`
`expected`	Optional ground-truth scores aligned with `predicted`; entries may be `None` when unavailable. TYPE: `Optional[Sequence[Optional[float]]]` DEFAULT: `None`
`num_bins`	Number of histogram/calibration bins over `[0, 1]`. TYPE: `int` DEFAULT: `DEFAULT_NUM_BINS`

Functions¶

histogram ¶

histogram() -> List[Tuple[float, float, int]]

Return the score histogram as (low, high, count) per bin.

calibration_curve ¶

calibration_curve() -> List[Tuple[float, float, int]]

Mean expected score per predicted-score bin.

RETURNS	DESCRIPTION
`List[Tuple[float, float, int]]`	`(bin_center, mean_expected, n)` for each non-empty bin. A
`List[Tuple[float, float, int]]`	well-calibrated judge has `mean_expected` close to
`List[Tuple[float, float, int]]`	`bin_center`. Empty when no expected scores were provided.

bucket_spread ¶

bucket_spread() -> Dict[str, Dict[str, float]]

Predicted-score mean/std grouped by expected-score bucket.

Buckets are low [0, 1/3), medium [1/3, 2/3) and high [2/3, 1], showing whether the judge separates examples that humans scored low, medium and high. Empty when no expected scores.

flags ¶

flags() -> List[str]

Human-readable warnings about judge pathologies.

print_summary ¶

print_summary() -> None

Print a text summary of the score distribution and any flags.

plot ¶

plot()

Plot the score histogram and calibration curve with matplotlib.

RETURNS	DESCRIPTION
	The created `matplotlib` figure.

RAISES	DESCRIPTION
`ImportError`	If `matplotlib` is not installed.

ScoreDistributionAnalyzer ¶

Run a feedback function over a golden set and analyze its scores.

PARAMETER	DESCRIPTION
`feedback_fn`	The feedback function to evaluate. Called once per golden example and expected to return a score in `[0, 1]` (a trailing metadata value or a `dict` of scores is also accepted). TYPE: `Callable[..., Any]`
`golden_set`	A list of dicts, each with `query`, `expected_response` and, optionally, `expected_score`. TYPE: `List[Dict[str, Any]]`
`num_bins`	Number of histogram/calibration bins over `[0, 1]`. TYPE: `int` DEFAULT: `DEFAULT_NUM_BINS`
`args_fn`	Optional function mapping a golden row to the positional arguments passed to `feedback_fn`. Defaults to `(row["query"], row["expected_response"])`. TYPE: `Optional[Callable[[Dict[str, Any]], Tuple]]` DEFAULT: `None`

Functions¶

run ¶

run() -> ScoreDistributionReport

Run the feedback function over the golden set.

RETURNS	DESCRIPTION
`ScoreDistributionReport`	A populated
`ScoreDistributionReport`	ScoreDistributionReport.

RAISES	DESCRIPTION
`ValueError`	If the feedback function produced no scores.