trulens.feedback.jury

trulens.feedback.jury

LLM Jury — ensemble multiple LLM judges into a single feedback callable.

A single LLM judge is noisy and subject to intra-model bias. Ensembling a panel of diverse judges (a "jury") improves reliability, reduces bias, and can be cheaper when smaller models are used.

Classes

Jury

Ensemble multiple LLM judges into a single feedback callable.

Jury wraps N provider instances, calls the same named method on each in parallel, and aggregates their scores using a configurable strategy. Because Jury.__call__ exposes the same parameter names as the underlying provider method, it plugs directly into Metric(implementation=jury) — no changes to Metric, Selector, or the evaluation pipeline are needed.

__call__ always returns (score, {"reason": ...}), matching the _with_cot_reasons convention so per-juror breakdowns flow into FeedbackCall.meta["reason"] and are visible in OTEL spans and the dashboard without any UI changes.

PARAMETER	DESCRIPTION
jurors	Non-empty list of LLMProvider instances. TYPE: list[Any]
method	Name of the feedback method to call on each juror, e.g. "relevance" or "groundedness_measure_with_cot_reasons". TYPE: str
aggregation	How to combine individual juror scores. Accepts a strategy name ("mean", "median", "trimmed_mean", "majority_vote", "weighted_mean") or any Callable[[list[float]], float]. Defaults to "mean". TYPE: str | Callable[[list[float]], float] DEFAULT: 'mean'
weights	Per-juror weights for "weighted_mean". Must have the same length as jurors. When a juror fails its weight is redistributed proportionally among the successful ones. TYPE: list[float] | None DEFAULT: None
threshold	Binarisation threshold for "majority_vote". Scores >= threshold count as a positive vote. Defaults to 0.5. On an exact tie falls back to median. TYPE: float DEFAULT: 0.5
max_workers	Maximum parallel threads. Defaults to len(jurors). TYPE: int | None DEFAULT: None

Example::

from trulens.core import Metric
from trulens.feedback.jury import Jury
from trulens.providers.openai import OpenAI
from trulens.providers.litellm import LiteLLM

jury = Jury(
    jurors=[
        OpenAI(model_engine="gpt-4o-mini"),
        OpenAI(model_engine="gpt-4.1-mini"),
        LiteLLM(model_engine="anthropic/claude-3-haiku-20240307"),
    ],
    method="relevance",
    aggregation="median",
)
m = Metric(implementation=jury, name="Jury Relevance").on_input().on_output()

Functions

__call__

__call__(
    *args: Any, **kwargs: Any
) -> tuple[float, dict[str, Any]]

Evaluate the same arguments in parallel across all jurors.

Always returns (score, {"reason": ...}), matching the _with_cot_reasons convention. Per-juror scores and any CoT explanations are embedded in the reason string so they appear in OTEL spans and the dashboard automatically.