Benchmark and Improve an LLM Judge¶
This notebook applies TruLens' LLM judge alignment workflow to a small labeled relevance dataset. In the rendered documentation, see LLM Judge Alignment for the full protocol.
It compares a baseline rubric with a stricter rubric, diagnoses each judge with AlignmentReport, and shows how to compare the same rubric across models. Every reported score comes from a configured provider. The notebook never generates simulated benchmark results.
Use a representative human-labeled dataset for real decisions. The rows below are intentionally small so the API flow is easy to adapt.
1. Install and configure¶
pip install trulens-benchmark trulens-providers-openai
export OPENAI_API_KEY="..."
Provider constructors read credentials from their normal environment or client configuration. Do not put keys in the notebook.
from trulens.benchmark import AlignmentReport
from trulens.benchmark.criteria_ab_test import CriteriaABTest
from trulens.benchmark.cross_model_alignment import CrossModelAlignment
from trulens.providers.openai import OpenAI
2. Define labeled development data¶
Each row contains the inputs passed to provider.relevance and a human score normalized to [0, 1]. Keep validation and held-out rows in separate datasets when tuning a production judge.
golden_set = [
{
"query": "How do I reset my password?",
"expected_response": "Use the reset link on the sign-in page.",
"expected_score": 1.0,
},
{
"query": "How do I reset my password?",
"expected_response": "Contact sales for enterprise pricing.",
"expected_score": 0.0,
},
{
"query": "Can I export my data?",
"expected_response": "You can export records as CSV from Settings.",
"expected_score": 1.0,
},
{
"query": "Can I export my data?",
"expected_response": "The product includes reporting features.",
"expected_score": 0.5,
},
]
3. Compare two rubric configurations¶
Keep the model and evaluated rows fixed. The kwargs dictionary is the only difference between these variants.
provider = OpenAI(model_engine="gpt-4o-mini")
strict_criteria = """
A response is relevant when it directly answers the user's request.
Score topical but incomplete responses below fully responsive answers.
Do not reward fluency or extra detail that does not answer the request.
"""
comparison = CriteriaABTest(
golden_set=golden_set,
variant_a={"fn": provider.relevance, "name": "default"},
variant_b={
"fn": provider.relevance,
"name": "strict-rubric",
"kwargs": {"criteria": strict_criteria},
},
)
comparison_report = comparison.run()
comparison_report.print_comparison()
The permutation p-value printed by CriteriaABTest measures raw score shift. Use the ground-truth metrics to choose a candidate, and do not interpret score drift alone as improved alignment.
4. Diagnose each variant¶
AlignmentReport accepts scores that have already been computed. This keeps judge execution separate from diagnostics and lets you cache real provider outputs for reproducibility.
true_labels = [row["expected_score"] for row in golden_set]
examples = [
{"query": row["query"], "response": row["expected_response"]}
for row in golden_set
]
reports = {
comparison_report.name_a: AlignmentReport(
predicted_scores=comparison_report.scores_a,
true_labels=true_labels,
examples=examples,
threshold=0.5,
thresholds=[0.3, 0.5, 0.7],
n_bins=4,
),
comparison_report.name_b: AlignmentReport(
predicted_scores=comparison_report.scores_b,
true_labels=true_labels,
examples=examples,
threshold=0.5,
thresholds=[0.3, 0.5, 0.7],
n_bins=4,
),
}
for name, report in reports.items():
print(f"\n{name}")
report.print_summary()
Inspect report.to_dataframe()["worst_misses"] before revising the rubric. Recheck confusion matrices at every production threshold; lower MAE does not guarantee better pass/fail decisions.
5. Compare judge models¶
Keep the method, rubric, and rows fixed when replacing the model. CrossModelAlignment reports pairwise agreement and each judge's agreement with ground truth.
judge_models = [
{
"provider": OpenAI(model_engine="gpt-4o-mini"),
"name": "gpt-4o-mini",
},
{
"provider": OpenAI(model_engine="gpt-4.1-mini"),
"name": "gpt-4.1-mini",
},
]
model_comparison = CrossModelAlignment(
judges=judge_models,
feedback_method="relevance",
golden_set=golden_set,
)
model_report = model_comparison.run()
model_report.print_matrix()
6. Make the decision on held-out data¶
Select a candidate using development and validation data, then freeze the complete configuration before scoring held-out rows. Record the provider, model, method, criteria, instructions, examples, output space, generation settings, dataset version, and threshold.
For a larger comparison with repeated trials, observed latency, and actual provider cost, see Comparing OpenAI Models for Evaluation.