trulens.benchmark.cross_model_alignment¶

trulens.benchmark.cross_model_alignment ¶

Cross-model alignment for feedback functions.

When you switch the model behind a judge (llama-3.3-70b to llama-3.1-8b, or one provider to another) the scores can shift in ways a single accuracy number hides. CrossModelAlignment runs the same feedback method with several judges over one golden set and reports how much they agree: pairwise Spearman correlation, mean absolute difference and score-shift bias between every pair, plus each judge's agreement with the ground truth, and a plain recommendation of which judges are interchangeable and which are outliers.

The Spearman, Kendall and MAE metrics reuse GroundTruthAggregator so there is a single source of truth for them rather than a second implementation.

Example

from trulens.benchmark.cross_model_alignment import CrossModelAlignment
from trulens.providers.litellm import LiteLLM

judge_a = LiteLLM(model_engine="groq/llama-3.3-70b-versatile")
judge_b = LiteLLM(model_engine="groq/llama-3.1-8b-instant")
alignment = CrossModelAlignment(
    judges=[
        {"provider": judge_a, "name": "70b"},
        {"provider": judge_b, "name": "8b"},
    ],
    feedback_method="relevance",
    golden_set=my_golden_set,
)
report = alignment.run()
report.print_matrix()
report.plot_heatmap()

Classes¶

CrossModelAlignmentReport ¶

Pairwise agreement between judges plus per-judge ground-truth metrics.

PARAMETER	DESCRIPTION
`names`	Judge names, one per score sequence. TYPE: `List[str]`
`scores`	Per-judge score sequences, all aligned and equal length. TYPE: `List[List[float]]`
`expected`	Optional ground-truth scores aligned with `scores`. TYPE: `Optional[List[float]]` DEFAULT: `None`
`interchangeable_spearman`	Pairs whose Spearman correlation is at least this are reported as interchangeable. TYPE: `float` DEFAULT: `INTERCHANGEABLE_SPEARMAN`
`outlier_mean_spearman`	A judge whose mean correlation with the others is below this is reported as an outlier. TYPE: `float` DEFAULT: `OUTLIER_MEAN_SPEARMAN`

Functions¶

ground_truth_metrics ¶

ground_truth_metrics() -> Dict[str, Dict[str, float]]

Per-judge agreement with the ground truth (mae, spearman, kendall).

Empty when no expected scores were provided.

recommendations ¶

recommendations() -> List[str]

Plain-language notes on interchangeable judges and outliers.

print_matrix ¶

print_matrix() -> None

Print the pairwise agreement matrices and recommendations.

plot_heatmap ¶

plot_heatmap()

Plot the pairwise Spearman matrix as a heatmap.

RETURNS	DESCRIPTION
	The created `matplotlib` figure.

RAISES	DESCRIPTION
`ImportError`	If `matplotlib` is not installed.

CrossModelAlignment ¶

Run one feedback method with several judges and compare their scores.

PARAMETER	DESCRIPTION
`judges`	A list of dicts, each with a `provider` (a feedback provider instance) and a `name`. TYPE: `List[Dict[str, Any]]`
`feedback_method`	The provider method to call as the judge, e.g. `"relevance"`. TYPE: `str`
`golden_set`	A list of dicts, each with `query`, `expected_response` and, optionally, `expected_score`. TYPE: `List[Dict[str, Any]]`
`args_fn`	Optional function mapping a golden row to the positional arguments passed to each judge. Defaults to `(row["query"], row["expected_response"])`. TYPE: `Optional[Callable[[Dict[str, Any]], Tuple]]` DEFAULT: `None`
`interchangeable_spearman`	Pairs whose Spearman correlation is at least this are reported as interchangeable. TYPE: `float` DEFAULT: `INTERCHANGEABLE_SPEARMAN`
`outlier_mean_spearman`	A judge whose mean correlation with the others is below this is reported as an outlier. TYPE: `float` DEFAULT: `OUTLIER_MEAN_SPEARMAN`
`skip_warn_fraction`	Warn if more than this fraction of rows are skipped because a judge errored on them. TYPE: `float` DEFAULT: `0.2`

Functions¶

run ¶

run() -> CrossModelAlignmentReport

Score the golden set with every judge and build the report.

Rows on which any judge errors are skipped so all judges stay aligned.

RETURNS	DESCRIPTION
`CrossModelAlignmentReport`	A populated
`CrossModelAlignmentReport`	CrossModelAlignmentReport.

RAISES	DESCRIPTION
`ValueError`	If no row was scored by every judge.