trulens.feedback.jury¶

trulens.feedback.jury ¶

LLM Jury — ensemble multiple LLM judges into a single feedback callable.

A single LLM judge is noisy and subject to intra-model bias. Ensembling a panel of diverse judges (a "jury") improves reliability, reduces bias, and can be cheaper when smaller models are used.

A jury can also repeat one judge (see :meth:Jury.repeated). At non-zero temperature the same judge, given identical inputs, does not always return the same verdict, so repeating it measures run-to-run variance that a single draw hides.

Every result carries a reliability summary over the juror scores: their spread, and for a pass/fail view the flip rate and outcome entropy. The reliability metrics follow "The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation" (arXiv:2606.13685), which measured pairwise judge verdicts flipping on average 13.6 percent of the time across repeated identical evaluations.

Classes¶

Jury ¶

Ensemble multiple LLM judges into a single feedback callable.

Jury wraps N provider instances, calls the same named method on each in parallel, and aggregates their scores using a configurable strategy. Because Jury.__call__ exposes the same parameter names as the underlying provider method, it plugs directly into Metric(implementation=jury) — no changes to Metric, Selector, or the evaluation pipeline are needed.

__call__ always returns (score, {"reason": ..., "reliability.*": ...}), matching the _with_cot_reasons convention so per-juror breakdowns flow into FeedbackCall.meta["reason"] and are visible in OTEL spans and the dashboard without any UI changes. The reliability keys are flat so each one lands on the eval span as its own typed attribute:

reliability.n_scores: jurors that returned a score.
reliability.scores: those scores, in juror order.
reliability.score_std: population standard deviation of the scores.
reliability.flip_rate: share of scores that disagree with the majority pass/fail verdict at threshold, 1 - max(counts) / n. This is the flip rate defined in arXiv:2606.13685, from 0.0 (unanimous) to 0.5 (even split).
reliability.outcome_entropy: entropy in bits of the pass/fail split, from 0.0 (unanimous) to 1.0 (even split).
reliability.temperature: the temperature passed to the jurors, or None when none was passed.

A juror counts as failed when it raises or when its judge reply carried no parseable score, which the providers report as :data:~trulens.feedback.llm_provider.UNPARSABLE_SCORE (-1.0). Either way it gets no vote. Only a jury with no surviving juror raises.

PARAMETER	DESCRIPTION
`jurors`	Non-empty list of `LLMProvider` instances. TYPE: `list[Any]`
`method`	Name of the feedback method to call on each juror, e.g. `"relevance"` or `"groundedness_measure_with_cot_reasons"`. TYPE: `str`
`aggregation`	How to combine individual juror scores. Accepts a strategy name (`"mean"`, `"median"`, `"trimmed_mean"`, `"majority_vote"`, `"weighted_mean"`) or any `Callable[[list[float]], float]`. Defaults to `"mean"`. TYPE: `str \| Callable[[list[float]], float]` DEFAULT: `'mean'`
`weights`	Per-juror weights for `"weighted_mean"`. Must have the same length as jurors. When a juror fails its weight is redistributed proportionally among the successful ones. TYPE: `list[float] \| None` DEFAULT: `None`
`threshold`	Binarisation threshold for `"majority_vote"` and for the pass/fail view behind `reliability.flip_rate` and `reliability.outcome_entropy`. Scores >= threshold count as a positive vote. Defaults to `0.5`. On an exact `"majority_vote"` tie falls back to median. TYPE: `float` DEFAULT: `0.5`
`max_workers`	Maximum parallel threads. Defaults to `len(jurors)`. TYPE: `int \| None` DEFAULT: `None`

Example::

from trulens.core import Metric
from trulens.feedback.jury import Jury
from trulens.providers.openai import OpenAI
from trulens.providers.litellm import LiteLLM

jury = Jury(
    jurors=[
        OpenAI(model_engine="gpt-4o-mini"),
        OpenAI(model_engine="gpt-4.1-mini"),
        LiteLLM(model_engine="anthropic/claude-3-haiku-20240307"),
    ],
    method="relevance",
    aggregation="median",
)
m = Metric(implementation=jury, name="Jury Relevance").on_input().on_output()

Functions¶

repeated `classmethod` ¶

repeated(
    judge: Any,
    method: str,
    n_trials: int = 5,
    aggregation: (
        str | Callable[[list[float]], float]
    ) = "mean",
    *,
    threshold: float = 0.5,
    max_workers: int | None = None
) -> Jury

Run one judge n_trials times and aggregate the scores.

Repeating a single judge measures its run-to-run variance, which a single draw hides. Judge methods and Metric both default to temperature=0.0; set a non-zero temperature on the Metric to sample distinct verdicts, and it is passed to every trial. At temperature 0 the trials are not sampled, so reliability.* then reflects only nondeterminism in the serving stack. A nonzero flip rate is still a real signal, but a zero does not show the judge is stable, and the jury logs a warning once. reliability.temperature records the temperature the trials ran at.

With n_trials >= 2 the trials are labeled model[0], model[1], ... in the reason text. arXiv:2606.13685 found about 11 trials are needed for a majority verdict to match a 50-trial reference with 95 percent probability, so raise n_trials when the verdict matters more than the cost.

Example::

from trulens.core import Metric
from trulens.feedback import Jury
from trulens.providers.openai import OpenAI

judge = Jury.repeated(
    OpenAI(model_engine="gpt-4o-mini"), method="relevance", n_trials=5
)
metric = Metric(
    implementation=judge, name="Relevance (5 trials)", temperature=0.7
).on_input().on_output()

call ¶

__call__(
    *args: Any, **kwargs: Any
) -> tuple[float, dict[str, Any]]

Evaluate the same arguments in parallel across all jurors.

Always returns (score, {"reason": ..., "reliability.*": ...}), matching the _with_cot_reasons convention. Per-juror scores and any CoT explanations are embedded in the reason string so they appear in OTEL spans and the dashboard automatically. The reliability.* keys are described on the class.

trulens.feedback.jury¶