LLM Jury: Ensemble Evaluation with Multiple Judges¶

A single LLM judge is noisy and subject to intra-model bias. Research shows that ensembling a panel of diverse judges (a "jury") substantially improves evaluation reliability and can be cheaper when smaller models are used.

Jury is a drop-in implementation for Metric: because its __call__ exposes the same parameter names as the underlying provider method, no changes to Metric, Selector, or the evaluation pipeline are needed.

Jury always returns (score, {"reason": ...}), matching the _with_cot_reasons convention — per-juror breakdowns appear in the dashboard automatically with no UI changes.

This notebook demonstrates:

1. Setting up a 3-model jury for relevance evaluation.
2. Comparing jury scores vs. single-judge scores on a small dataset.
3. Inspecting per-juror score breakdowns via meta["reason"].
4. Using a weighted jury with alignment-derived weights.

In [ ]:

# !pip install trulens trulens-providers-openai trulens-providers-litellm

# !pip install trulens trulens-providers-openai trulens-providers-litellm

Setup¶

In [ ]:

import os

os.environ["OPENAI_API_KEY"] = "sk-..."  # replace with your key

import os os.environ["OPENAI_API_KEY"] = "sk-..." # replace with your key

1. Providers¶

In [ ]:

from trulens.providers.openai import OpenAI

provider_mini = OpenAI(model_engine="gpt-4o-mini")
provider_41   = OpenAI(model_engine="gpt-4.1-mini")
# provider_haiku = LiteLLM(model_engine="anthropic/claude-3-haiku-20240307")

from trulens.providers.openai import OpenAI provider_mini = OpenAI(model_engine="gpt-4o-mini") provider_41 = OpenAI(model_engine="gpt-4.1-mini") # provider_haiku = LiteLLM(model_engine="anthropic/claude-3-haiku-20240307")

2. Build a 3-model jury for relevance¶

Jury always returns (score, {"reason": ...}). The reason string contains the aggregation header, each juror's score, and any CoT explanations — it flows into OTEL spans and the dashboard automatically.

In [ ]:

from trulens.feedback.jury import Jury

jury = Jury(
    jurors=[provider_mini, provider_41],
    method="relevance",
    aggregation="median",
)

# Jury always returns (score, meta) — no extra flag needed.
score, meta = jury(
    prompt="What is TruLens?",
    response="TruLens is an open-source library for evaluating LLM applications.",
)
print(f"Jury score:\n{score:.3f}")
print(f"\nPer-juror breakdown (meta['reason']):\n{meta['reason']}")

from trulens.feedback.jury import Jury jury = Jury( jurors=[provider_mini, provider_41], method="relevance", aggregation="median", ) # Jury always returns (score, meta) — no extra flag needed. score, meta = jury( prompt="What is TruLens?", response="TruLens is an open-source library for evaluating LLM applications.", ) print(f"Jury score:\n{score:.3f}") print(f"\nPer-juror breakdown (meta['reason']):\n{meta['reason']}")

3. Wire the jury into Metric¶

In [ ]:

from trulens.core import Metric

m_jury = (
    Metric(implementation=jury, name="Jury Relevance")
    .on_input()
    .on_output()
)

# Single-judge baseline for comparison
m_single = (
    Metric(implementation=provider_mini.relevance, name="Single Relevance")
    .on_input()
    .on_output()
)

from trulens.core import Metric m_jury = ( Metric(implementation=jury, name="Jury Relevance") .on_input() .on_output() ) # Single-judge baseline for comparison m_single = ( Metric(implementation=provider_mini.relevance, name="Single Relevance") .on_input() .on_output() )

4. Compare jury vs. single judge on a small dataset¶

In [ ]:

dataset = [
    {
        "prompt": "What does TruLens do?",
        "response": "TruLens evaluates LLM-based applications with feedback functions.",
    },
    {
        "prompt": "What is the capital of France?",
        "response": "The capital of France is Paris.",
    },
    {
        "prompt": "Explain gradient descent.",
        "response": "I like pizza.",  # intentionally off-topic
    },
]

results = []
for row in dataset:
    jury_score, meta = jury(**row)
    single_score = provider_mini.relevance(**row)
    results.append({
        "prompt": row["prompt"][:40],
        "jury": round(jury_score, 3),
        "single": round(single_score, 3),
        "reason": meta["reason"],
    })

for r in results:
    print(f"Prompt : {r['prompt']}")
    print(f"Jury   : {r['jury']}  Single: {r['single']}")
    print(f"Reason :\n{r['reason']}")
    print()

dataset = [ { "prompt": "What does TruLens do?", "response": "TruLens evaluates LLM-based applications with feedback functions.", }, { "prompt": "What is the capital of France?", "response": "The capital of France is Paris.", }, { "prompt": "Explain gradient descent.", "response": "I like pizza.", # intentionally off-topic }, ] results = [] for row in dataset: jury_score, meta = jury(**row) single_score = provider_mini.relevance(**row) results.append({ "prompt": row["prompt"][:40], "jury": round(jury_score, 3), "single": round(single_score, 3), "reason": meta["reason"], }) for r in results: print(f"Prompt : {r['prompt']}") print(f"Jury : {r['jury']} Single: {r['single']}") print(f"Reason :\n{r['reason']}") print()

5. Weighted jury from benchmark alignment scores¶

If you have benchmark-derived alignment scores for each judge, use weighted_mean to give more trusted judges a higher weight.

In [ ]:

weighted_jury = Jury(
    jurors=[provider_mini, provider_41],
    method="relevance",
    aggregation="weighted_mean",
    weights=[0.6, 0.4],  # gpt-4o-mini scored higher on our alignment benchmark
)

score, meta = weighted_jury(
    prompt="What is TruLens?",
    response="An evaluation framework for LLM apps.",
)
print(f"Weighted jury score: {score:.3f}")
print(f"\nBreakdown:\n{meta['reason']}")

weighted_jury = Jury( jurors=[provider_mini, provider_41], method="relevance", aggregation="weighted_mean", weights=[0.6, 0.4], # gpt-4o-mini scored higher on our alignment benchmark ) score, meta = weighted_jury( prompt="What is TruLens?", response="An evaluation framework for LLM apps.", ) print(f"Weighted jury score: {score:.3f}") print(f"\nBreakdown:\n{meta['reason']}")

6. Use with TruApp¶

Because Jury plugs directly into Metric, you can pass jury metrics to TruApp exactly like any other feedback function.

In [ ]:

# from trulens.apps.langchain import TruChain
# tru_app = TruChain(my_chain, feedbacks=[m_jury])
# with tru_app:
#     my_chain.invoke("What is TruLens?")

# from trulens.apps.langchain import TruChain # tru_app = TruChain(my_chain, feedbacks=[m_jury]) # with tru_app: # my_chain.invoke("What is TruLens?")