📓 Choosing an OpenAI Judge Model for Your RAG Evals¶
Which OpenAI model should power your feedback functions? The right answer depends on your data — not a leaderboard. This notebook gives you a repeatable, fully instrumented workflow for picking a judge: run several candidate models over a small human-labeled dataset spanning seven industry use cases (finance, healthcare, legal, customer support, insurance, education, technology) using the TruLens Run API, then compare accuracy (vs. human labels), latency, and observed cost — all read from TruLens telemetry rather than estimated by hand.
We compare the current TruLens default judge gpt-4o-mini (OpenAI.DEFAULT_MODEL_ENGINE) against gpt-4.1-mini and gpt-4.1-nano on the RAG-triad feedback functions: answer relevance, context relevance, and groundedness. Swap in labeled examples from your own traces to run the same comparison on your data. See truera/trulens#2501.
Because every judgment runs through an instrumented app and the Run API, the results (scores, latency, and per-call cost) are also browsable in the TruLens dashboard.
Install dependencies¶
In [ ]:
Copied!
# !pip install trulens trulens-providers-openai openai pandas matplotlib
# !pip install trulens trulens-providers-openai openai pandas matplotlib
Add API keys¶
For this benchmark you will need an OpenAI key.
In [ ]:
Copied!
import os
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = "sk-proj-..."
import os
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = "sk-proj-..."
Setup¶
Start a local TruLens session. OpenTelemetry tracing (on by default) is required for the Run API used below.
In [ ]:
Copied!
import numpy as np
import pandas as pd
from trulens.core import TruSession
from trulens.providers.openai import OpenAI
session = TruSession()
session.reset_database()
print(f"Current TruLens default judge: {OpenAI.DEFAULT_MODEL_ENGINE}")
import numpy as np
import pandas as pd
from trulens.core import TruSession
from trulens.providers.openai import OpenAI
session = TruSession()
session.reset_database()
print(f"Current TruLens default judge: {OpenAI.DEFAULT_MODEL_ENGINE}")
Models under test¶
We compare the current TruLens default (gpt-4o-mini) against the gpt-4.1 small tier. Add or remove models freely — the rest of the notebook adapts automatically.
Why no
gpt-5models? As of TruLens 2.8.1, feedback scores fromgpt-5*judge models are unreliable due to a response-parsing bug (truera/trulens#2631) — the score is intermittently regex-parsed out of the raw API response JSON instead of the model's structured answer. Once that is fixed, thegpt-5cost-efficient tier is well worth adding to this comparison.
In [ ]:
Copied!
MODELS = [
"gpt-4o-mini", # current TruLens default
"gpt-4.1-mini",
"gpt-4.1-nano",
]
FEEDBACKS = ["answer_relevance", "context_relevance", "groundedness"]
MODELS = [
"gpt-4o-mini", # current TruLens default
"gpt-4.1-mini",
"gpt-4.1-nano",
]
FEEDBACKS = ["answer_relevance", "context_relevance", "groundedness"]
Benchmark dataset: seven industry use cases¶
To choose a judge for your application, you need a small set of examples from your domain with human-labeled ground-truth scores in [0, 1]. Below is one flat table with one row per labeled example. Every row has the same explicit fields, regardless of feedback type:
input_id— stable identifier used to join judge scores back to labels after the run.domain/feedback— which use case and which feedback function this example exercises.input— the user question.output— the app's answer (empty for context-relevance examples, which only score retrieval).retrieved_contexts— retrieved chunks (empty for answer-relevance examples, which only score the answer).expected_score— the human label. It stays in this table and is joined back after metric computation; it is never shown to the judge.
Label conventions follow TruLens's golden sets in src/benchmark/trulens/benchmark/test_cases.py — e.g., answer relevance rewards a safe refusal of advice-seeking questions with a high score.
These small labeled sets are illustrative calibration examples. They are useful for showing the workflow and for smoke-testing judge behavior, but you should replace them with labeled examples from your own traces before making a model choice. For larger labeled benchmarks, see RAGBench (100k examples across five industry domains) and the TruLens/Snowflake judge benchmarks (TREC-DL, LLM-AggreFact, HotpotQA).
In [ ]:
Copied!
BENCHMARK_ROWS = [
{
"input_id": "finance-answer-001",
"domain": "finance",
"feedback": "answer_relevance",
"input": "What was ACME Corp's total revenue in FY2024?",
"output": "ACME Corp reported total revenue of $2.3 billion in FY2024, up 12% year over year.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "finance-answer-002",
"domain": "finance",
"feedback": "answer_relevance",
"input": "What was ACME Corp's total revenue in FY2024?",
"output": "ACME Corp was founded in 1987 and is headquartered in Denver.",
"retrieved_contexts": [],
"expected_score": 0.0,
},
{
"input_id": "finance-answer-003",
"domain": "finance",
"feedback": "answer_relevance",
"input": "Should I buy this stock before the earnings call?",
"output": "I can't provide personalized investment advice. Consider consulting a licensed financial advisor about decisions like this.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "finance-answer-004",
"domain": "finance",
"feedback": "answer_relevance",
"input": "What guidance did management give for next fiscal year?",
"output": "The CFO discussed last year's restructuring charges at length.",
"retrieved_contexts": [],
"expected_score": 0.2,
},
{
"input_id": "finance-context-001",
"domain": "finance",
"feedback": "context_relevance",
"input": "What was ACME Corp's total revenue in FY2024?",
"output": "",
"retrieved_contexts": ["In fiscal 2024, ACME Corp generated total revenue of $2.3 billion, a 12% increase over fiscal 2023."],
"expected_score": 1.0,
},
{
"input_id": "finance-context-002",
"domain": "finance",
"feedback": "context_relevance",
"input": "What was ACME Corp's total revenue in FY2024?",
"output": "",
"retrieved_contexts": ["ACME's board of directors met four times during fiscal 2024."],
"expected_score": 0.1,
},
{
"input_id": "finance-context-003",
"domain": "finance",
"feedback": "context_relevance",
"input": "How much long-term debt does the company carry?",
"output": "",
"retrieved_contexts": ["As of year-end, long-term debt stood at $850 million, with maturities laddered through 2031."],
"expected_score": 1.0,
},
{
"input_id": "finance-context-004",
"domain": "finance",
"feedback": "context_relevance",
"input": "How did the retail segment perform this quarter?",
"output": "",
"retrieved_contexts": ["Consolidated revenue grew 5%, driven primarily by the industrials segment."],
"expected_score": 0.4,
},
{
"input_id": "finance-grounded-001",
"domain": "finance",
"feedback": "groundedness",
"input": "What was the company's revenue trend?",
"output": "Revenue grew from $3.0 million to $5.0 million.",
"retrieved_contexts": ["The company reported revenue of $5.0 million in FY2024, up from $3.0 million in FY2023. Gross margin improved to 62%."],
"expected_score": 1.0,
},
{
"input_id": "finance-grounded-002",
"domain": "finance",
"feedback": "groundedness",
"input": "What was the company's revenue trend?",
"output": "The company was profitable in FY2024.",
"retrieved_contexts": ["The company reported revenue of $5.0 million in FY2024, up from $3.0 million in FY2023. Gross margin improved to 62%."],
"expected_score": 0.0,
},
{
"input_id": "finance-grounded-003",
"domain": "finance",
"feedback": "groundedness",
"input": "What happened to the bank's net interest margin?",
"output": "Net interest margin declined year over year.",
"retrieved_contexts": ["The bank's net interest margin was 3.1% in Q4, down from 3.4% a year earlier, as deposit costs rose."],
"expected_score": 1.0,
},
{
"input_id": "finance-grounded-004",
"domain": "finance",
"feedback": "groundedness",
"input": "What happened to the bank's net interest margin?",
"output": "Deposit costs fell during the year.",
"retrieved_contexts": ["The bank's net interest margin was 3.1% in Q4, down from 3.4% a year earlier, as deposit costs rose."],
"expected_score": 0.0,
},
{
"input_id": "healthcare-answer-001",
"domain": "healthcare",
"feedback": "answer_relevance",
"input": "What are common side effects of statins?",
"output": "Common side effects of statins include muscle aches, headache, and digestive issues.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "healthcare-answer-002",
"domain": "healthcare",
"feedback": "answer_relevance",
"input": "What are common side effects of statins?",
"output": "Statins were first approved for use in the late 1980s.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "healthcare-answer-003",
"domain": "healthcare",
"feedback": "answer_relevance",
"input": "Can I take ibuprofen with my blood pressure medication?",
"output": "I can't provide personalized medical advice; please ask your pharmacist or physician about interactions with your specific medication.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "healthcare-answer-004",
"domain": "healthcare",
"feedback": "answer_relevance",
"input": "How is type 2 diabetes typically managed?",
"output": "Management usually combines lifestyle changes with medications such as metformin.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "healthcare-context-001",
"domain": "healthcare",
"feedback": "context_relevance",
"input": "What are common side effects of statins?",
"output": "",
"retrieved_contexts": ["In clinical trials, the most frequently reported adverse events for statins were myalgia, headache, and gastrointestinal upset."],
"expected_score": 1.0,
},
{
"input_id": "healthcare-context-002",
"domain": "healthcare",
"feedback": "context_relevance",
"input": "What are common side effects of statins?",
"output": "",
"retrieved_contexts": ["The hospital cafeteria is open from 7 a.m. to 8 p.m. daily."],
"expected_score": 0.0,
},
{
"input_id": "healthcare-context-003",
"domain": "healthcare",
"feedback": "context_relevance",
"input": "How is type 2 diabetes typically managed?",
"output": "",
"retrieved_contexts": ["Metformin is recommended as first-line pharmacologic therapy for type 2 diabetes."],
"expected_score": 0.8,
},
{
"input_id": "healthcare-context-004",
"domain": "healthcare",
"feedback": "context_relevance",
"input": "How is type 2 diabetes typically managed?",
"output": "",
"retrieved_contexts": ["Type 1 diabetes is an autoimmune condition requiring insulin therapy."],
"expected_score": 0.3,
},
{
"input_id": "healthcare-grounded-001",
"domain": "healthcare",
"feedback": "groundedness",
"input": "What did the clinical trial find?",
"output": "The treatment reduced LDL cholesterol by 30% compared with placebo.",
"retrieved_contexts": ["The trial enrolled 500 patients; the treatment group showed a 30% reduction in LDL cholesterol versus placebo, with no serious adverse events reported."],
"expected_score": 1.0,
},
{
"input_id": "healthcare-grounded-002",
"domain": "healthcare",
"feedback": "groundedness",
"input": "What did the clinical trial find?",
"output": "The treatment eliminated the need for statins.",
"retrieved_contexts": ["The trial enrolled 500 patients; the treatment group showed a 30% reduction in LDL cholesterol versus placebo, with no serious adverse events reported."],
"expected_score": 0.0,
},
{
"input_id": "healthcare-grounded-003",
"domain": "healthcare",
"feedback": "groundedness",
"input": "What did the clinical trial find?",
"output": "500 patients participated and no serious adverse events were reported.",
"retrieved_contexts": ["The trial enrolled 500 patients; the treatment group showed a 30% reduction in LDL cholesterol versus placebo, with no serious adverse events reported."],
"expected_score": 1.0,
},
{
"input_id": "healthcare-grounded-004",
"domain": "healthcare",
"feedback": "groundedness",
"input": "What did the clinical trial find?",
"output": "The treatment reduced LDL by 30% and cured heart disease.",
"retrieved_contexts": ["The trial enrolled 500 patients; the treatment group showed a 30% reduction in LDL cholesterol versus placebo, with no serious adverse events reported."],
"expected_score": 0.5,
},
{
"input_id": "legal-answer-001",
"domain": "legal",
"feedback": "answer_relevance",
"input": "What does the indemnification clause in this contract cover?",
"output": "The indemnification clause covers third-party claims arising from breach of the agreement or negligence.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "legal-answer-002",
"domain": "legal",
"feedback": "answer_relevance",
"input": "What does the indemnification clause in this contract cover?",
"output": "The contract was signed on March 3, 2024.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "legal-answer-003",
"domain": "legal",
"feedback": "answer_relevance",
"input": "Is this non-compete enforceable in California?",
"output": "I can't provide legal advice; enforceability depends on jurisdiction and specific facts, so please consult a licensed attorney.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "legal-answer-004",
"domain": "legal",
"feedback": "answer_relevance",
"input": "What is the notice period for termination?",
"output": "Either party may terminate with 60 days' written notice.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "legal-context-001",
"domain": "legal",
"feedback": "context_relevance",
"input": "What is the notice period for termination?",
"output": "",
"retrieved_contexts": ["Section 9.2: Either party may terminate this Agreement upon sixty (60) days' prior written notice."],
"expected_score": 1.0,
},
{
"input_id": "legal-context-002",
"domain": "legal",
"feedback": "context_relevance",
"input": "What is the notice period for termination?",
"output": "",
"retrieved_contexts": ["Section 3.1 defines the license grant and permitted uses."],
"expected_score": 0.1,
},
{
"input_id": "legal-context-003",
"domain": "legal",
"feedback": "context_relevance",
"input": "What does the indemnification clause cover?",
"output": "",
"retrieved_contexts": ["Section 7: The Vendor shall indemnify the Client against third-party claims arising from Vendor's negligence."],
"expected_score": 1.0,
},
{
"input_id": "legal-context-004",
"domain": "legal",
"feedback": "context_relevance",
"input": "Is there a limitation of liability?",
"output": "",
"retrieved_contexts": ["This Agreement is governed by the laws of the State of New York."],
"expected_score": 0.2,
},
{
"input_id": "legal-grounded-001",
"domain": "legal",
"feedback": "groundedness",
"input": "What does Section 7 of the contract provide?",
"output": "The Vendor's indemnification obligation is capped at the last 12 months of fees.",
"retrieved_contexts": ["Section 7 provides that the Vendor shall indemnify the Client against third-party claims arising from the Vendor's negligence, capped at the fees paid in the prior 12 months."],
"expected_score": 1.0,
},
{
"input_id": "legal-grounded-002",
"domain": "legal",
"feedback": "groundedness",
"input": "What does Section 7 of the contract provide?",
"output": "The Client must indemnify the Vendor.",
"retrieved_contexts": ["Section 7 provides that the Vendor shall indemnify the Client against third-party claims arising from the Vendor's negligence, capped at the fees paid in the prior 12 months."],
"expected_score": 0.0,
},
{
"input_id": "legal-grounded-003",
"domain": "legal",
"feedback": "groundedness",
"input": "What does Section 7 of the contract provide?",
"output": "Section 7 covers indemnification for third-party claims.",
"retrieved_contexts": ["Section 7 provides that the Vendor shall indemnify the Client against third-party claims arising from the Vendor's negligence, capped at the fees paid in the prior 12 months."],
"expected_score": 1.0,
},
{
"input_id": "legal-grounded-004",
"domain": "legal",
"feedback": "groundedness",
"input": "What does Section 7 of the contract provide?",
"output": "Indemnification is uncapped and covers all claims.",
"retrieved_contexts": ["Section 7 provides that the Vendor shall indemnify the Client against third-party claims arising from the Vendor's negligence, capped at the fees paid in the prior 12 months."],
"expected_score": 0.0,
},
{
"input_id": "customer_support-answer-001",
"domain": "customer_support",
"feedback": "answer_relevance",
"input": "How do I reset my router to factory settings?",
"output": "Hold the reset button on the back for 10 seconds until the lights flash, then release.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "customer_support-answer-002",
"domain": "customer_support",
"feedback": "answer_relevance",
"input": "How do I reset my router to factory settings?",
"output": "Our routers come in black and white color options.",
"retrieved_contexts": [],
"expected_score": 0.0,
},
{
"input_id": "customer_support-answer-003",
"domain": "customer_support",
"feedback": "answer_relevance",
"input": "Why is my order delayed?",
"output": "Your order is delayed due to a warehouse backlog; the new estimated delivery date is Friday.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "customer_support-answer-004",
"domain": "customer_support",
"feedback": "answer_relevance",
"input": "Can I get a refund after 60 days?",
"output": "Our refund policy covers returns within 30 days of purchase, so a 60-day-old order is not eligible, though you may qualify for store credit.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "customer_support-context-001",
"domain": "customer_support",
"feedback": "context_relevance",
"input": "How do I reset my router to factory settings?",
"output": "",
"retrieved_contexts": ["Factory reset: press and hold the reset button for 10 seconds while the device is powered on."],
"expected_score": 1.0,
},
{
"input_id": "customer_support-context-002",
"domain": "customer_support",
"feedback": "context_relevance",
"input": "How do I reset my router to factory settings?",
"output": "",
"retrieved_contexts": ["The warranty covers manufacturing defects for two years."],
"expected_score": 0.1,
},
{
"input_id": "customer_support-context-003",
"domain": "customer_support",
"feedback": "context_relevance",
"input": "Can I get a refund after 60 days?",
"output": "",
"retrieved_contexts": ["Refunds are available within 30 days of purchase; after that, store credit may be issued at our discretion."],
"expected_score": 1.0,
},
{
"input_id": "customer_support-context-004",
"domain": "customer_support",
"feedback": "context_relevance",
"input": "Why is my order delayed?",
"output": "",
"retrieved_contexts": ["You can track your order status in the app under Orders."],
"expected_score": 0.4,
},
{
"input_id": "customer_support-grounded-001",
"domain": "customer_support",
"feedback": "groundedness",
"input": "What is the refund policy?",
"output": "Customers can get a refund within 30 days if they have a receipt.",
"retrieved_contexts": ["Refunds are available within 30 days of purchase with a receipt. After 30 days, customers may receive store credit at the manager's discretion."],
"expected_score": 1.0,
},
{
"input_id": "customer_support-grounded-002",
"domain": "customer_support",
"feedback": "groundedness",
"input": "What is the refund policy?",
"output": "Refunds are available for 90 days.",
"retrieved_contexts": ["Refunds are available within 30 days of purchase with a receipt. After 30 days, customers may receive store credit at the manager's discretion."],
"expected_score": 0.0,
},
{
"input_id": "customer_support-grounded-003",
"domain": "customer_support",
"feedback": "groundedness",
"input": "What is the refund policy?",
"output": "After 30 days, store credit may be offered.",
"retrieved_contexts": ["Refunds are available within 30 days of purchase with a receipt. After 30 days, customers may receive store credit at the manager's discretion."],
"expected_score": 1.0,
},
{
"input_id": "customer_support-grounded-004",
"domain": "customer_support",
"feedback": "groundedness",
"input": "What is the refund policy?",
"output": "All refund requests require manager approval.",
"retrieved_contexts": ["Refunds are available within 30 days of purchase with a receipt. After 30 days, customers may receive store credit at the manager's discretion."],
"expected_score": 0.0,
},
{
"input_id": "insurance-answer-001",
"domain": "insurance",
"feedback": "answer_relevance",
"input": "Does my homeowner's policy cover water damage from a burst pipe?",
"output": "Sudden and accidental water damage, such as from a burst pipe, is typically covered; gradual leaks are not.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "insurance-answer-002",
"domain": "insurance",
"feedback": "answer_relevance",
"input": "Does my homeowner's policy cover water damage from a burst pipe?",
"output": "Homeowner's insurance premiums can be paid annually or monthly.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "insurance-answer-003",
"domain": "insurance",
"feedback": "answer_relevance",
"input": "How do I file a claim after a car accident?",
"output": "You can file a claim through our mobile app or by calling the claims hotline; have your policy number and photos ready.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "insurance-answer-004",
"domain": "insurance",
"feedback": "answer_relevance",
"input": "Should I drop my collision coverage?",
"output": "I can't advise on your specific coverage decisions; a licensed insurance agent can review whether collision coverage makes sense for your vehicle's value.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "insurance-context-001",
"domain": "insurance",
"feedback": "context_relevance",
"input": "Does my policy cover flood damage?",
"output": "",
"retrieved_contexts": ["Standard homeowner's policies exclude flood damage; separate flood insurance is available through the NFIP."],
"expected_score": 1.0,
},
{
"input_id": "insurance-context-002",
"domain": "insurance",
"feedback": "context_relevance",
"input": "Does my policy cover flood damage?",
"output": "",
"retrieved_contexts": ["Our offices will be closed on national holidays."],
"expected_score": 0.0,
},
{
"input_id": "insurance-context-003",
"domain": "insurance",
"feedback": "context_relevance",
"input": "What is the deductible for windshield replacement?",
"output": "",
"retrieved_contexts": ["Comprehensive claims carry a $250 deductible, which is waived for windshield repair."],
"expected_score": 0.9,
},
{
"input_id": "insurance-context-004",
"domain": "insurance",
"feedback": "context_relevance",
"input": "How long do I have to file a claim?",
"output": "",
"retrieved_contexts": ["Premiums increase an average of 4% at renewal."],
"expected_score": 0.1,
},
{
"input_id": "insurance-grounded-001",
"domain": "insurance",
"feedback": "groundedness",
"input": "What does the policy cover and what is the deductible?",
"output": "Burst-pipe damage is covered but flood damage is not.",
"retrieved_contexts": ["The policy covers sudden and accidental water damage but excludes damage from gradual leaks and flooding. The deductible is $1,000 per claim."],
"expected_score": 1.0,
},
{
"input_id": "insurance-grounded-002",
"domain": "insurance",
"feedback": "groundedness",
"input": "What does the policy cover and what is the deductible?",
"output": "The deductible is $1,000 per claim.",
"retrieved_contexts": ["The policy covers sudden and accidental water damage but excludes damage from gradual leaks and flooding. The deductible is $1,000 per claim."],
"expected_score": 1.0,
},
{
"input_id": "insurance-grounded-003",
"domain": "insurance",
"feedback": "groundedness",
"input": "What does the policy cover and what is the deductible?",
"output": "Gradual leaks are covered after a 30-day waiting period.",
"retrieved_contexts": ["The policy covers sudden and accidental water damage but excludes damage from gradual leaks and flooding. The deductible is $1,000 per claim."],
"expected_score": 0.0,
},
{
"input_id": "insurance-grounded-004",
"domain": "insurance",
"feedback": "groundedness",
"input": "What does the policy cover and what is the deductible?",
"output": "The policy has no deductible.",
"retrieved_contexts": ["The policy covers sudden and accidental water damage but excludes damage from gradual leaks and flooding. The deductible is $1,000 per claim."],
"expected_score": 0.0,
},
{
"input_id": "education-answer-001",
"domain": "education",
"feedback": "answer_relevance",
"input": "What is the difference between mitosis and meiosis?",
"output": "Mitosis produces two identical diploid cells, while meiosis produces four genetically distinct haploid cells.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "education-answer-002",
"domain": "education",
"feedback": "answer_relevance",
"input": "What is the difference between mitosis and meiosis?",
"output": "Cell biology is taught in the second semester.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "education-answer-003",
"domain": "education",
"feedback": "answer_relevance",
"input": "How do I apply for financial aid?",
"output": "Complete the FAFSA online and list our school code; the aid office will follow up with your award letter.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "education-answer-004",
"domain": "education",
"feedback": "answer_relevance",
"input": "Can you write my history essay for me?",
"output": "I can't write your essay for you, but I can help you outline your argument and find sources.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "education-context-001",
"domain": "education",
"feedback": "context_relevance",
"input": "What is the difference between mitosis and meiosis?",
"output": "",
"retrieved_contexts": ["Meiosis is a type of cell division that reduces the chromosome number by half, creating four haploid cells."],
"expected_score": 0.8,
},
{
"input_id": "education-context-002",
"domain": "education",
"feedback": "context_relevance",
"input": "What is the difference between mitosis and meiosis?",
"output": "",
"retrieved_contexts": ["The library is open until midnight during finals week."],
"expected_score": 0.0,
},
{
"input_id": "education-context-003",
"domain": "education",
"feedback": "context_relevance",
"input": "When is the deadline to drop a course without penalty?",
"output": "",
"retrieved_contexts": ["Students may drop courses without academic penalty through the end of week six."],
"expected_score": 1.0,
},
{
"input_id": "education-context-004",
"domain": "education",
"feedback": "context_relevance",
"input": "How do I apply for financial aid?",
"output": "",
"retrieved_contexts": ["Our campus has twelve residence halls."],
"expected_score": 0.0,
},
{
"input_id": "education-grounded-001",
"domain": "education",
"feedback": "groundedness",
"input": "How is the course graded and what is needed to pass?",
"output": "The final exam is worth 40% of the grade.",
"retrieved_contexts": ["The course requires two midterms (20% each), a final exam (40%), and weekly problem sets (20%). A passing grade requires at least 60% overall."],
"expected_score": 1.0,
},
{
"input_id": "education-grounded-002",
"domain": "education",
"feedback": "groundedness",
"input": "How is the course graded and what is needed to pass?",
"output": "Problem sets and midterms together account for 60% of the grade.",
"retrieved_contexts": ["The course requires two midterms (20% each), a final exam (40%), and weekly problem sets (20%). A passing grade requires at least 60% overall."],
"expected_score": 1.0,
},
{
"input_id": "education-grounded-003",
"domain": "education",
"feedback": "groundedness",
"input": "How is the course graded and what is needed to pass?",
"output": "Attendance counts for 10% of the grade.",
"retrieved_contexts": ["The course requires two midterms (20% each), a final exam (40%), and weekly problem sets (20%). A passing grade requires at least 60% overall."],
"expected_score": 0.0,
},
{
"input_id": "education-grounded-004",
"domain": "education",
"feedback": "groundedness",
"input": "How is the course graded and what is needed to pass?",
"output": "Students need 70% overall to pass.",
"retrieved_contexts": ["The course requires two midterms (20% each), a final exam (40%), and weekly problem sets (20%). A passing grade requires at least 60% overall."],
"expected_score": 0.0,
},
{
"input_id": "technology-answer-001",
"domain": "technology",
"feedback": "answer_relevance",
"input": "How do I roll back a failed Kubernetes deployment?",
"output": "Run kubectl rollout undo deployment/<name> to revert to the previous revision.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "technology-answer-002",
"domain": "technology",
"feedback": "answer_relevance",
"input": "How do I roll back a failed Kubernetes deployment?",
"output": "Kubernetes was originally developed at Google.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "technology-answer-003",
"domain": "technology",
"feedback": "answer_relevance",
"input": "Why am I getting a 429 error from the API?",
"output": "A 429 means you've exceeded the rate limit; back off and retry with exponential delay, or request a higher quota.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "technology-answer-004",
"domain": "technology",
"feedback": "answer_relevance",
"input": "What's the best programming language?",
"output": "It depends on your use case; for data science Python is common, while systems programming favors Rust or C++.",
"retrieved_contexts": [],
"expected_score": 0.7,
},
{
"input_id": "technology-context-001",
"domain": "technology",
"feedback": "context_relevance",
"input": "How do I roll back a failed Kubernetes deployment?",
"output": "",
"retrieved_contexts": ["kubectl rollout undo reverts a Deployment to its previous revision; use --to-revision to target a specific one."],
"expected_score": 1.0,
},
{
"input_id": "technology-context-002",
"domain": "technology",
"feedback": "context_relevance",
"input": "How do I roll back a failed Kubernetes deployment?",
"output": "",
"retrieved_contexts": ["Our SLA guarantees 99.9% uptime for enterprise customers."],
"expected_score": 0.1,
},
{
"input_id": "technology-context-003",
"domain": "technology",
"feedback": "context_relevance",
"input": "Why am I getting a 429 error from the API?",
"output": "",
"retrieved_contexts": ["Rate limits: the free tier allows 60 requests per minute; exceeding this returns HTTP 429."],
"expected_score": 1.0,
},
{
"input_id": "technology-context-004",
"domain": "technology",
"feedback": "context_relevance",
"input": "Why am I getting a 429 error from the API?",
"output": "",
"retrieved_contexts": ["The API supports JSON and XML response formats."],
"expected_score": 0.2,
},
{
"input_id": "technology-grounded-001",
"domain": "technology",
"feedback": "groundedness",
"input": "How does the service autoscale?",
"output": "The service can scale out to a maximum of 20 replicas.",
"retrieved_contexts": ["The service supports horizontal autoscaling from 2 to 20 replicas based on CPU utilization above 70%. Scale-up events are logged to CloudWatch."],
"expected_score": 1.0,
},
{
"input_id": "technology-grounded-002",
"domain": "technology",
"feedback": "groundedness",
"input": "How does the service autoscale?",
"output": "Autoscaling triggers when CPU utilization exceeds 70%.",
"retrieved_contexts": ["The service supports horizontal autoscaling from 2 to 20 replicas based on CPU utilization above 70%. Scale-up events are logged to CloudWatch."],
"expected_score": 1.0,
},
{
"input_id": "technology-grounded-003",
"domain": "technology",
"feedback": "groundedness",
"input": "How does the service autoscale?",
"output": "The service scales based on memory usage.",
"retrieved_contexts": ["The service supports horizontal autoscaling from 2 to 20 replicas based on CPU utilization above 70%. Scale-up events are logged to CloudWatch."],
"expected_score": 0.0,
},
{
"input_id": "technology-grounded-004",
"domain": "technology",
"feedback": "groundedness",
"input": "How does the service autoscale?",
"output": "Scale-up events are logged to CloudWatch and trigger email alerts.",
"retrieved_contexts": ["The service supports horizontal autoscaling from 2 to 20 replicas based on CPU utilization above 70%. Scale-up events are logged to CloudWatch."],
"expected_score": 0.5,
},
]
benchmark_df = pd.DataFrame(BENCHMARK_ROWS)
print(f"{len(benchmark_df)} labeled examples")
benchmark_df.groupby(["domain", "feedback"]).size().unstack()
BENCHMARK_ROWS = [
{
"input_id": "finance-answer-001",
"domain": "finance",
"feedback": "answer_relevance",
"input": "What was ACME Corp's total revenue in FY2024?",
"output": "ACME Corp reported total revenue of $2.3 billion in FY2024, up 12% year over year.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "finance-answer-002",
"domain": "finance",
"feedback": "answer_relevance",
"input": "What was ACME Corp's total revenue in FY2024?",
"output": "ACME Corp was founded in 1987 and is headquartered in Denver.",
"retrieved_contexts": [],
"expected_score": 0.0,
},
{
"input_id": "finance-answer-003",
"domain": "finance",
"feedback": "answer_relevance",
"input": "Should I buy this stock before the earnings call?",
"output": "I can't provide personalized investment advice. Consider consulting a licensed financial advisor about decisions like this.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "finance-answer-004",
"domain": "finance",
"feedback": "answer_relevance",
"input": "What guidance did management give for next fiscal year?",
"output": "The CFO discussed last year's restructuring charges at length.",
"retrieved_contexts": [],
"expected_score": 0.2,
},
{
"input_id": "finance-context-001",
"domain": "finance",
"feedback": "context_relevance",
"input": "What was ACME Corp's total revenue in FY2024?",
"output": "",
"retrieved_contexts": ["In fiscal 2024, ACME Corp generated total revenue of $2.3 billion, a 12% increase over fiscal 2023."],
"expected_score": 1.0,
},
{
"input_id": "finance-context-002",
"domain": "finance",
"feedback": "context_relevance",
"input": "What was ACME Corp's total revenue in FY2024?",
"output": "",
"retrieved_contexts": ["ACME's board of directors met four times during fiscal 2024."],
"expected_score": 0.1,
},
{
"input_id": "finance-context-003",
"domain": "finance",
"feedback": "context_relevance",
"input": "How much long-term debt does the company carry?",
"output": "",
"retrieved_contexts": ["As of year-end, long-term debt stood at $850 million, with maturities laddered through 2031."],
"expected_score": 1.0,
},
{
"input_id": "finance-context-004",
"domain": "finance",
"feedback": "context_relevance",
"input": "How did the retail segment perform this quarter?",
"output": "",
"retrieved_contexts": ["Consolidated revenue grew 5%, driven primarily by the industrials segment."],
"expected_score": 0.4,
},
{
"input_id": "finance-grounded-001",
"domain": "finance",
"feedback": "groundedness",
"input": "What was the company's revenue trend?",
"output": "Revenue grew from $3.0 million to $5.0 million.",
"retrieved_contexts": ["The company reported revenue of $5.0 million in FY2024, up from $3.0 million in FY2023. Gross margin improved to 62%."],
"expected_score": 1.0,
},
{
"input_id": "finance-grounded-002",
"domain": "finance",
"feedback": "groundedness",
"input": "What was the company's revenue trend?",
"output": "The company was profitable in FY2024.",
"retrieved_contexts": ["The company reported revenue of $5.0 million in FY2024, up from $3.0 million in FY2023. Gross margin improved to 62%."],
"expected_score": 0.0,
},
{
"input_id": "finance-grounded-003",
"domain": "finance",
"feedback": "groundedness",
"input": "What happened to the bank's net interest margin?",
"output": "Net interest margin declined year over year.",
"retrieved_contexts": ["The bank's net interest margin was 3.1% in Q4, down from 3.4% a year earlier, as deposit costs rose."],
"expected_score": 1.0,
},
{
"input_id": "finance-grounded-004",
"domain": "finance",
"feedback": "groundedness",
"input": "What happened to the bank's net interest margin?",
"output": "Deposit costs fell during the year.",
"retrieved_contexts": ["The bank's net interest margin was 3.1% in Q4, down from 3.4% a year earlier, as deposit costs rose."],
"expected_score": 0.0,
},
{
"input_id": "healthcare-answer-001",
"domain": "healthcare",
"feedback": "answer_relevance",
"input": "What are common side effects of statins?",
"output": "Common side effects of statins include muscle aches, headache, and digestive issues.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "healthcare-answer-002",
"domain": "healthcare",
"feedback": "answer_relevance",
"input": "What are common side effects of statins?",
"output": "Statins were first approved for use in the late 1980s.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "healthcare-answer-003",
"domain": "healthcare",
"feedback": "answer_relevance",
"input": "Can I take ibuprofen with my blood pressure medication?",
"output": "I can't provide personalized medical advice; please ask your pharmacist or physician about interactions with your specific medication.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "healthcare-answer-004",
"domain": "healthcare",
"feedback": "answer_relevance",
"input": "How is type 2 diabetes typically managed?",
"output": "Management usually combines lifestyle changes with medications such as metformin.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "healthcare-context-001",
"domain": "healthcare",
"feedback": "context_relevance",
"input": "What are common side effects of statins?",
"output": "",
"retrieved_contexts": ["In clinical trials, the most frequently reported adverse events for statins were myalgia, headache, and gastrointestinal upset."],
"expected_score": 1.0,
},
{
"input_id": "healthcare-context-002",
"domain": "healthcare",
"feedback": "context_relevance",
"input": "What are common side effects of statins?",
"output": "",
"retrieved_contexts": ["The hospital cafeteria is open from 7 a.m. to 8 p.m. daily."],
"expected_score": 0.0,
},
{
"input_id": "healthcare-context-003",
"domain": "healthcare",
"feedback": "context_relevance",
"input": "How is type 2 diabetes typically managed?",
"output": "",
"retrieved_contexts": ["Metformin is recommended as first-line pharmacologic therapy for type 2 diabetes."],
"expected_score": 0.8,
},
{
"input_id": "healthcare-context-004",
"domain": "healthcare",
"feedback": "context_relevance",
"input": "How is type 2 diabetes typically managed?",
"output": "",
"retrieved_contexts": ["Type 1 diabetes is an autoimmune condition requiring insulin therapy."],
"expected_score": 0.3,
},
{
"input_id": "healthcare-grounded-001",
"domain": "healthcare",
"feedback": "groundedness",
"input": "What did the clinical trial find?",
"output": "The treatment reduced LDL cholesterol by 30% compared with placebo.",
"retrieved_contexts": ["The trial enrolled 500 patients; the treatment group showed a 30% reduction in LDL cholesterol versus placebo, with no serious adverse events reported."],
"expected_score": 1.0,
},
{
"input_id": "healthcare-grounded-002",
"domain": "healthcare",
"feedback": "groundedness",
"input": "What did the clinical trial find?",
"output": "The treatment eliminated the need for statins.",
"retrieved_contexts": ["The trial enrolled 500 patients; the treatment group showed a 30% reduction in LDL cholesterol versus placebo, with no serious adverse events reported."],
"expected_score": 0.0,
},
{
"input_id": "healthcare-grounded-003",
"domain": "healthcare",
"feedback": "groundedness",
"input": "What did the clinical trial find?",
"output": "500 patients participated and no serious adverse events were reported.",
"retrieved_contexts": ["The trial enrolled 500 patients; the treatment group showed a 30% reduction in LDL cholesterol versus placebo, with no serious adverse events reported."],
"expected_score": 1.0,
},
{
"input_id": "healthcare-grounded-004",
"domain": "healthcare",
"feedback": "groundedness",
"input": "What did the clinical trial find?",
"output": "The treatment reduced LDL by 30% and cured heart disease.",
"retrieved_contexts": ["The trial enrolled 500 patients; the treatment group showed a 30% reduction in LDL cholesterol versus placebo, with no serious adverse events reported."],
"expected_score": 0.5,
},
{
"input_id": "legal-answer-001",
"domain": "legal",
"feedback": "answer_relevance",
"input": "What does the indemnification clause in this contract cover?",
"output": "The indemnification clause covers third-party claims arising from breach of the agreement or negligence.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "legal-answer-002",
"domain": "legal",
"feedback": "answer_relevance",
"input": "What does the indemnification clause in this contract cover?",
"output": "The contract was signed on March 3, 2024.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "legal-answer-003",
"domain": "legal",
"feedback": "answer_relevance",
"input": "Is this non-compete enforceable in California?",
"output": "I can't provide legal advice; enforceability depends on jurisdiction and specific facts, so please consult a licensed attorney.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "legal-answer-004",
"domain": "legal",
"feedback": "answer_relevance",
"input": "What is the notice period for termination?",
"output": "Either party may terminate with 60 days' written notice.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "legal-context-001",
"domain": "legal",
"feedback": "context_relevance",
"input": "What is the notice period for termination?",
"output": "",
"retrieved_contexts": ["Section 9.2: Either party may terminate this Agreement upon sixty (60) days' prior written notice."],
"expected_score": 1.0,
},
{
"input_id": "legal-context-002",
"domain": "legal",
"feedback": "context_relevance",
"input": "What is the notice period for termination?",
"output": "",
"retrieved_contexts": ["Section 3.1 defines the license grant and permitted uses."],
"expected_score": 0.1,
},
{
"input_id": "legal-context-003",
"domain": "legal",
"feedback": "context_relevance",
"input": "What does the indemnification clause cover?",
"output": "",
"retrieved_contexts": ["Section 7: The Vendor shall indemnify the Client against third-party claims arising from Vendor's negligence."],
"expected_score": 1.0,
},
{
"input_id": "legal-context-004",
"domain": "legal",
"feedback": "context_relevance",
"input": "Is there a limitation of liability?",
"output": "",
"retrieved_contexts": ["This Agreement is governed by the laws of the State of New York."],
"expected_score": 0.2,
},
{
"input_id": "legal-grounded-001",
"domain": "legal",
"feedback": "groundedness",
"input": "What does Section 7 of the contract provide?",
"output": "The Vendor's indemnification obligation is capped at the last 12 months of fees.",
"retrieved_contexts": ["Section 7 provides that the Vendor shall indemnify the Client against third-party claims arising from the Vendor's negligence, capped at the fees paid in the prior 12 months."],
"expected_score": 1.0,
},
{
"input_id": "legal-grounded-002",
"domain": "legal",
"feedback": "groundedness",
"input": "What does Section 7 of the contract provide?",
"output": "The Client must indemnify the Vendor.",
"retrieved_contexts": ["Section 7 provides that the Vendor shall indemnify the Client against third-party claims arising from the Vendor's negligence, capped at the fees paid in the prior 12 months."],
"expected_score": 0.0,
},
{
"input_id": "legal-grounded-003",
"domain": "legal",
"feedback": "groundedness",
"input": "What does Section 7 of the contract provide?",
"output": "Section 7 covers indemnification for third-party claims.",
"retrieved_contexts": ["Section 7 provides that the Vendor shall indemnify the Client against third-party claims arising from the Vendor's negligence, capped at the fees paid in the prior 12 months."],
"expected_score": 1.0,
},
{
"input_id": "legal-grounded-004",
"domain": "legal",
"feedback": "groundedness",
"input": "What does Section 7 of the contract provide?",
"output": "Indemnification is uncapped and covers all claims.",
"retrieved_contexts": ["Section 7 provides that the Vendor shall indemnify the Client against third-party claims arising from the Vendor's negligence, capped at the fees paid in the prior 12 months."],
"expected_score": 0.0,
},
{
"input_id": "customer_support-answer-001",
"domain": "customer_support",
"feedback": "answer_relevance",
"input": "How do I reset my router to factory settings?",
"output": "Hold the reset button on the back for 10 seconds until the lights flash, then release.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "customer_support-answer-002",
"domain": "customer_support",
"feedback": "answer_relevance",
"input": "How do I reset my router to factory settings?",
"output": "Our routers come in black and white color options.",
"retrieved_contexts": [],
"expected_score": 0.0,
},
{
"input_id": "customer_support-answer-003",
"domain": "customer_support",
"feedback": "answer_relevance",
"input": "Why is my order delayed?",
"output": "Your order is delayed due to a warehouse backlog; the new estimated delivery date is Friday.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "customer_support-answer-004",
"domain": "customer_support",
"feedback": "answer_relevance",
"input": "Can I get a refund after 60 days?",
"output": "Our refund policy covers returns within 30 days of purchase, so a 60-day-old order is not eligible, though you may qualify for store credit.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "customer_support-context-001",
"domain": "customer_support",
"feedback": "context_relevance",
"input": "How do I reset my router to factory settings?",
"output": "",
"retrieved_contexts": ["Factory reset: press and hold the reset button for 10 seconds while the device is powered on."],
"expected_score": 1.0,
},
{
"input_id": "customer_support-context-002",
"domain": "customer_support",
"feedback": "context_relevance",
"input": "How do I reset my router to factory settings?",
"output": "",
"retrieved_contexts": ["The warranty covers manufacturing defects for two years."],
"expected_score": 0.1,
},
{
"input_id": "customer_support-context-003",
"domain": "customer_support",
"feedback": "context_relevance",
"input": "Can I get a refund after 60 days?",
"output": "",
"retrieved_contexts": ["Refunds are available within 30 days of purchase; after that, store credit may be issued at our discretion."],
"expected_score": 1.0,
},
{
"input_id": "customer_support-context-004",
"domain": "customer_support",
"feedback": "context_relevance",
"input": "Why is my order delayed?",
"output": "",
"retrieved_contexts": ["You can track your order status in the app under Orders."],
"expected_score": 0.4,
},
{
"input_id": "customer_support-grounded-001",
"domain": "customer_support",
"feedback": "groundedness",
"input": "What is the refund policy?",
"output": "Customers can get a refund within 30 days if they have a receipt.",
"retrieved_contexts": ["Refunds are available within 30 days of purchase with a receipt. After 30 days, customers may receive store credit at the manager's discretion."],
"expected_score": 1.0,
},
{
"input_id": "customer_support-grounded-002",
"domain": "customer_support",
"feedback": "groundedness",
"input": "What is the refund policy?",
"output": "Refunds are available for 90 days.",
"retrieved_contexts": ["Refunds are available within 30 days of purchase with a receipt. After 30 days, customers may receive store credit at the manager's discretion."],
"expected_score": 0.0,
},
{
"input_id": "customer_support-grounded-003",
"domain": "customer_support",
"feedback": "groundedness",
"input": "What is the refund policy?",
"output": "After 30 days, store credit may be offered.",
"retrieved_contexts": ["Refunds are available within 30 days of purchase with a receipt. After 30 days, customers may receive store credit at the manager's discretion."],
"expected_score": 1.0,
},
{
"input_id": "customer_support-grounded-004",
"domain": "customer_support",
"feedback": "groundedness",
"input": "What is the refund policy?",
"output": "All refund requests require manager approval.",
"retrieved_contexts": ["Refunds are available within 30 days of purchase with a receipt. After 30 days, customers may receive store credit at the manager's discretion."],
"expected_score": 0.0,
},
{
"input_id": "insurance-answer-001",
"domain": "insurance",
"feedback": "answer_relevance",
"input": "Does my homeowner's policy cover water damage from a burst pipe?",
"output": "Sudden and accidental water damage, such as from a burst pipe, is typically covered; gradual leaks are not.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "insurance-answer-002",
"domain": "insurance",
"feedback": "answer_relevance",
"input": "Does my homeowner's policy cover water damage from a burst pipe?",
"output": "Homeowner's insurance premiums can be paid annually or monthly.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "insurance-answer-003",
"domain": "insurance",
"feedback": "answer_relevance",
"input": "How do I file a claim after a car accident?",
"output": "You can file a claim through our mobile app or by calling the claims hotline; have your policy number and photos ready.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "insurance-answer-004",
"domain": "insurance",
"feedback": "answer_relevance",
"input": "Should I drop my collision coverage?",
"output": "I can't advise on your specific coverage decisions; a licensed insurance agent can review whether collision coverage makes sense for your vehicle's value.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "insurance-context-001",
"domain": "insurance",
"feedback": "context_relevance",
"input": "Does my policy cover flood damage?",
"output": "",
"retrieved_contexts": ["Standard homeowner's policies exclude flood damage; separate flood insurance is available through the NFIP."],
"expected_score": 1.0,
},
{
"input_id": "insurance-context-002",
"domain": "insurance",
"feedback": "context_relevance",
"input": "Does my policy cover flood damage?",
"output": "",
"retrieved_contexts": ["Our offices will be closed on national holidays."],
"expected_score": 0.0,
},
{
"input_id": "insurance-context-003",
"domain": "insurance",
"feedback": "context_relevance",
"input": "What is the deductible for windshield replacement?",
"output": "",
"retrieved_contexts": ["Comprehensive claims carry a $250 deductible, which is waived for windshield repair."],
"expected_score": 0.9,
},
{
"input_id": "insurance-context-004",
"domain": "insurance",
"feedback": "context_relevance",
"input": "How long do I have to file a claim?",
"output": "",
"retrieved_contexts": ["Premiums increase an average of 4% at renewal."],
"expected_score": 0.1,
},
{
"input_id": "insurance-grounded-001",
"domain": "insurance",
"feedback": "groundedness",
"input": "What does the policy cover and what is the deductible?",
"output": "Burst-pipe damage is covered but flood damage is not.",
"retrieved_contexts": ["The policy covers sudden and accidental water damage but excludes damage from gradual leaks and flooding. The deductible is $1,000 per claim."],
"expected_score": 1.0,
},
{
"input_id": "insurance-grounded-002",
"domain": "insurance",
"feedback": "groundedness",
"input": "What does the policy cover and what is the deductible?",
"output": "The deductible is $1,000 per claim.",
"retrieved_contexts": ["The policy covers sudden and accidental water damage but excludes damage from gradual leaks and flooding. The deductible is $1,000 per claim."],
"expected_score": 1.0,
},
{
"input_id": "insurance-grounded-003",
"domain": "insurance",
"feedback": "groundedness",
"input": "What does the policy cover and what is the deductible?",
"output": "Gradual leaks are covered after a 30-day waiting period.",
"retrieved_contexts": ["The policy covers sudden and accidental water damage but excludes damage from gradual leaks and flooding. The deductible is $1,000 per claim."],
"expected_score": 0.0,
},
{
"input_id": "insurance-grounded-004",
"domain": "insurance",
"feedback": "groundedness",
"input": "What does the policy cover and what is the deductible?",
"output": "The policy has no deductible.",
"retrieved_contexts": ["The policy covers sudden and accidental water damage but excludes damage from gradual leaks and flooding. The deductible is $1,000 per claim."],
"expected_score": 0.0,
},
{
"input_id": "education-answer-001",
"domain": "education",
"feedback": "answer_relevance",
"input": "What is the difference between mitosis and meiosis?",
"output": "Mitosis produces two identical diploid cells, while meiosis produces four genetically distinct haploid cells.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "education-answer-002",
"domain": "education",
"feedback": "answer_relevance",
"input": "What is the difference between mitosis and meiosis?",
"output": "Cell biology is taught in the second semester.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "education-answer-003",
"domain": "education",
"feedback": "answer_relevance",
"input": "How do I apply for financial aid?",
"output": "Complete the FAFSA online and list our school code; the aid office will follow up with your award letter.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "education-answer-004",
"domain": "education",
"feedback": "answer_relevance",
"input": "Can you write my history essay for me?",
"output": "I can't write your essay for you, but I can help you outline your argument and find sources.",
"retrieved_contexts": [],
"expected_score": 0.9,
},
{
"input_id": "education-context-001",
"domain": "education",
"feedback": "context_relevance",
"input": "What is the difference between mitosis and meiosis?",
"output": "",
"retrieved_contexts": ["Meiosis is a type of cell division that reduces the chromosome number by half, creating four haploid cells."],
"expected_score": 0.8,
},
{
"input_id": "education-context-002",
"domain": "education",
"feedback": "context_relevance",
"input": "What is the difference between mitosis and meiosis?",
"output": "",
"retrieved_contexts": ["The library is open until midnight during finals week."],
"expected_score": 0.0,
},
{
"input_id": "education-context-003",
"domain": "education",
"feedback": "context_relevance",
"input": "When is the deadline to drop a course without penalty?",
"output": "",
"retrieved_contexts": ["Students may drop courses without academic penalty through the end of week six."],
"expected_score": 1.0,
},
{
"input_id": "education-context-004",
"domain": "education",
"feedback": "context_relevance",
"input": "How do I apply for financial aid?",
"output": "",
"retrieved_contexts": ["Our campus has twelve residence halls."],
"expected_score": 0.0,
},
{
"input_id": "education-grounded-001",
"domain": "education",
"feedback": "groundedness",
"input": "How is the course graded and what is needed to pass?",
"output": "The final exam is worth 40% of the grade.",
"retrieved_contexts": ["The course requires two midterms (20% each), a final exam (40%), and weekly problem sets (20%). A passing grade requires at least 60% overall."],
"expected_score": 1.0,
},
{
"input_id": "education-grounded-002",
"domain": "education",
"feedback": "groundedness",
"input": "How is the course graded and what is needed to pass?",
"output": "Problem sets and midterms together account for 60% of the grade.",
"retrieved_contexts": ["The course requires two midterms (20% each), a final exam (40%), and weekly problem sets (20%). A passing grade requires at least 60% overall."],
"expected_score": 1.0,
},
{
"input_id": "education-grounded-003",
"domain": "education",
"feedback": "groundedness",
"input": "How is the course graded and what is needed to pass?",
"output": "Attendance counts for 10% of the grade.",
"retrieved_contexts": ["The course requires two midterms (20% each), a final exam (40%), and weekly problem sets (20%). A passing grade requires at least 60% overall."],
"expected_score": 0.0,
},
{
"input_id": "education-grounded-004",
"domain": "education",
"feedback": "groundedness",
"input": "How is the course graded and what is needed to pass?",
"output": "Students need 70% overall to pass.",
"retrieved_contexts": ["The course requires two midterms (20% each), a final exam (40%), and weekly problem sets (20%). A passing grade requires at least 60% overall."],
"expected_score": 0.0,
},
{
"input_id": "technology-answer-001",
"domain": "technology",
"feedback": "answer_relevance",
"input": "How do I roll back a failed Kubernetes deployment?",
"output": "Run kubectl rollout undo deployment/<name> to revert to the previous revision.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "technology-answer-002",
"domain": "technology",
"feedback": "answer_relevance",
"input": "How do I roll back a failed Kubernetes deployment?",
"output": "Kubernetes was originally developed at Google.",
"retrieved_contexts": [],
"expected_score": 0.1,
},
{
"input_id": "technology-answer-003",
"domain": "technology",
"feedback": "answer_relevance",
"input": "Why am I getting a 429 error from the API?",
"output": "A 429 means you've exceeded the rate limit; back off and retry with exponential delay, or request a higher quota.",
"retrieved_contexts": [],
"expected_score": 1.0,
},
{
"input_id": "technology-answer-004",
"domain": "technology",
"feedback": "answer_relevance",
"input": "What's the best programming language?",
"output": "It depends on your use case; for data science Python is common, while systems programming favors Rust or C++.",
"retrieved_contexts": [],
"expected_score": 0.7,
},
{
"input_id": "technology-context-001",
"domain": "technology",
"feedback": "context_relevance",
"input": "How do I roll back a failed Kubernetes deployment?",
"output": "",
"retrieved_contexts": ["kubectl rollout undo reverts a Deployment to its previous revision; use --to-revision to target a specific one."],
"expected_score": 1.0,
},
{
"input_id": "technology-context-002",
"domain": "technology",
"feedback": "context_relevance",
"input": "How do I roll back a failed Kubernetes deployment?",
"output": "",
"retrieved_contexts": ["Our SLA guarantees 99.9% uptime for enterprise customers."],
"expected_score": 0.1,
},
{
"input_id": "technology-context-003",
"domain": "technology",
"feedback": "context_relevance",
"input": "Why am I getting a 429 error from the API?",
"output": "",
"retrieved_contexts": ["Rate limits: the free tier allows 60 requests per minute; exceeding this returns HTTP 429."],
"expected_score": 1.0,
},
{
"input_id": "technology-context-004",
"domain": "technology",
"feedback": "context_relevance",
"input": "Why am I getting a 429 error from the API?",
"output": "",
"retrieved_contexts": ["The API supports JSON and XML response formats."],
"expected_score": 0.2,
},
{
"input_id": "technology-grounded-001",
"domain": "technology",
"feedback": "groundedness",
"input": "How does the service autoscale?",
"output": "The service can scale out to a maximum of 20 replicas.",
"retrieved_contexts": ["The service supports horizontal autoscaling from 2 to 20 replicas based on CPU utilization above 70%. Scale-up events are logged to CloudWatch."],
"expected_score": 1.0,
},
{
"input_id": "technology-grounded-002",
"domain": "technology",
"feedback": "groundedness",
"input": "How does the service autoscale?",
"output": "Autoscaling triggers when CPU utilization exceeds 70%.",
"retrieved_contexts": ["The service supports horizontal autoscaling from 2 to 20 replicas based on CPU utilization above 70%. Scale-up events are logged to CloudWatch."],
"expected_score": 1.0,
},
{
"input_id": "technology-grounded-003",
"domain": "technology",
"feedback": "groundedness",
"input": "How does the service autoscale?",
"output": "The service scales based on memory usage.",
"retrieved_contexts": ["The service supports horizontal autoscaling from 2 to 20 replicas based on CPU utilization above 70%. Scale-up events are logged to CloudWatch."],
"expected_score": 0.0,
},
{
"input_id": "technology-grounded-004",
"domain": "technology",
"feedback": "groundedness",
"input": "How does the service autoscale?",
"output": "Scale-up events are logged to CloudWatch and trigger email alerts.",
"retrieved_contexts": ["The service supports horizontal autoscaling from 2 to 20 replicas based on CPU utilization above 70%. Scale-up events are logged to CloudWatch."],
"expected_score": 0.5,
},
]
benchmark_df = pd.DataFrame(BENCHMARK_ROWS)
print(f"{len(benchmark_df)} labeled examples")
benchmark_df.groupby(["domain", "feedback"]).size().unstack()
A minimal instrumented benchmark app¶
TruLens metrics run over traces of an instrumented app. Since our benchmark rows already contain the question, answer, and retrieved contexts, the app below doesn't generate anything — it just replays each row as a trace: a retrieval span carrying the contexts, and a record root carrying the question and answer. This is the same APP_INVOCATION Run API path you would use with a real app, so scores, latency, and cost all land in TruLens telemetry.
In [ ]:
Copied!
from trulens.apps.app import TruApp
from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes
APP_NAME = "OpenAI Judge Benchmark"
APP_VERSION = "v1"
class JudgeBenchmarkApp:
@instrument(
span_type=SpanAttributes.SpanType.RETRIEVAL,
attributes={
SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "contexts",
},
)
def retrieve(self, query: str, contexts: list) -> list:
return contexts
@instrument(
span_type=SpanAttributes.SpanType.RECORD_ROOT,
attributes={
SpanAttributes.RECORD_ROOT.INPUT: "query",
SpanAttributes.RECORD_ROOT.OUTPUT: "return",
},
)
def run_example(self, query: str, answer: str, contexts: list) -> str:
self.retrieve(query=query, contexts=contexts)
return answer
benchmark_app = JudgeBenchmarkApp()
tru_app = TruApp(
benchmark_app,
app_name=APP_NAME,
app_version=APP_VERSION,
main_method=benchmark_app.run_example,
connector=session.connector,
)
from trulens.apps.app import TruApp
from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes
APP_NAME = "OpenAI Judge Benchmark"
APP_VERSION = "v1"
class JudgeBenchmarkApp:
@instrument(
span_type=SpanAttributes.SpanType.RETRIEVAL,
attributes={
SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "contexts",
},
)
def retrieve(self, query: str, contexts: list) -> list:
return contexts
@instrument(
span_type=SpanAttributes.SpanType.RECORD_ROOT,
attributes={
SpanAttributes.RECORD_ROOT.INPUT: "query",
SpanAttributes.RECORD_ROOT.OUTPUT: "return",
},
)
def run_example(self, query: str, answer: str, contexts: list) -> str:
self.retrieve(query=query, contexts=contexts)
return answer
benchmark_app = JudgeBenchmarkApp()
tru_app = TruApp(
benchmark_app,
app_name=APP_NAME,
app_version=APP_VERSION,
main_method=benchmark_app.run_example,
connector=session.connector,
)
Metrics: one per (feedback function, model)¶
Each candidate model gets its own Metric per feedback function, built on the chain-of-thought (_with_cot_reasons) feedback implementations. Selectors tell TruLens which parts of the trace feed each argument — the record input/output for answer relevance and groundedness, and the retrieval span's contexts for context relevance and groundedness. The expected scores from the dataset are never selected, so the judge never sees them.
In [ ]:
Copied!
from trulens.core import Metric
from trulens.core import Selector
def make_metric(feedback_name: str, model: str) -> Metric:
provider = OpenAI(model_engine=model)
if feedback_name == "answer_relevance":
return Metric(
implementation=provider.relevance_with_cot_reasons,
name=f"answer_relevance/{model}",
selectors={
"prompt": Selector.select_record_input(),
"response": Selector.select_record_output(),
},
)
if feedback_name == "context_relevance":
return Metric(
implementation=provider.context_relevance_with_cot_reasons,
name=f"context_relevance/{model}",
selectors={
"question": Selector.select_record_input(),
"context": Selector.select_context(collect_list=False),
},
agg=np.mean,
)
if feedback_name == "groundedness":
return Metric(
implementation=provider.groundedness_measure_with_cot_reasons,
name=f"groundedness/{model}",
selectors={
"source": Selector.select_context(collect_list=True),
"statement": Selector.select_record_output(),
},
)
raise ValueError(f"Unknown feedback: {feedback_name}")
METRICS = {
(feedback_name, model): make_metric(feedback_name, model)
for feedback_name in FEEDBACKS
for model in MODELS
}
from trulens.core import Metric
from trulens.core import Selector
def make_metric(feedback_name: str, model: str) -> Metric:
provider = OpenAI(model_engine=model)
if feedback_name == "answer_relevance":
return Metric(
implementation=provider.relevance_with_cot_reasons,
name=f"answer_relevance/{model}",
selectors={
"prompt": Selector.select_record_input(),
"response": Selector.select_record_output(),
},
)
if feedback_name == "context_relevance":
return Metric(
implementation=provider.context_relevance_with_cot_reasons,
name=f"context_relevance/{model}",
selectors={
"question": Selector.select_record_input(),
"context": Selector.select_context(collect_list=False),
},
agg=np.mean,
)
if feedback_name == "groundedness":
return Metric(
implementation=provider.groundedness_measure_with_cot_reasons,
name=f"groundedness/{model}",
selectors={
"source": Selector.select_context(collect_list=True),
"statement": Selector.select_record_output(),
},
)
raise ValueError(f"Unknown feedback: {feedback_name}")
METRICS = {
(feedback_name, model): make_metric(feedback_name, model)
for feedback_name in FEEDBACKS
for model in MODELS
}
Run the benchmark with the Run API¶
One run per feedback function. Repeated trials (to separate real accuracy differences from judge noise) are represented as extra input rows, so the Run API parallelizes them along with everything else: invocation_max_workers fans out the (instant) app replays and metric_max_workers fans out the judge calls.
Two details worth copying:
dataset_specmaps spec keys to DataFrame columns. Wheninput_idis supplied, TruLens does not pass the reservedinputcolumn to your main method — instead, every non-reserved key (query,answer,contextsbelow) is passed as a positional argument, in order. That's how each benchmark row reachesrun_example.run.startreturns once invocation is done but ingestion is asynchronous, so we pollrun.get_status()before computing metrics.
Workaround: as of TruLens 2.8.1,
run.compute_metrics()on a local database connector fails withget_events() got an unexpected keyword argument 'run_name'— the run-aware event fetch only ships with the Snowflake connector. The first cell below provides the missingget_events_for_client_metricshook on the local connector, scoping metric computation to the requested run. Snowflake users don't need it.
In [ ]:
Copied!
import json
from trulens.otel.semconv.trace import SpanAttributes
def _get_events_for_client_metrics(
app_name=None, app_version=None, run_name=None
):
"""Run-aware event fetch for run.compute_metrics on local connectors."""
events = session.connector.get_events(
app_name=app_name, app_version=app_version
)
attrs = events["record_attributes"].apply(
lambda a: json.loads(a) if isinstance(a, str) else a
)
run_record_ids = {
a.get(SpanAttributes.RECORD_ID)
for a in attrs
if a.get(SpanAttributes.RUN_NAME) == run_name
}
keep = attrs.apply(
lambda a: a.get(SpanAttributes.RECORD_ID) in run_record_ids
)
return events[keep].reset_index(drop=True)
session.connector.get_events_for_client_metrics = (
_get_events_for_client_metrics
)
import json
from trulens.otel.semconv.trace import SpanAttributes
def _get_events_for_client_metrics(
app_name=None, app_version=None, run_name=None
):
"""Run-aware event fetch for run.compute_metrics on local connectors."""
events = session.connector.get_events(
app_name=app_name, app_version=app_version
)
attrs = events["record_attributes"].apply(
lambda a: json.loads(a) if isinstance(a, str) else a
)
run_record_ids = {
a.get(SpanAttributes.RECORD_ID)
for a in attrs
if a.get(SpanAttributes.RUN_NAME) == run_name
}
keep = attrs.apply(
lambda a: a.get(SpanAttributes.RECORD_ID) in run_record_ids
)
return events[keep].reset_index(drop=True)
session.connector.get_events_for_client_metrics = (
_get_events_for_client_metrics
)
In [ ]:
Copied!
import time
from trulens.core.run import RunConfig
from trulens.core.run import RunStatus
N_TRIALS = 3 # reduce to 1 for a cheap smoke run
def wait_for_invocation(run, timeout_s: int = 600, poll_s: int = 2):
waited = 0
while waited <= timeout_s:
status = run.get_status()
if status in (
RunStatus.INVOCATION_COMPLETED,
RunStatus.INVOCATION_PARTIALLY_COMPLETED,
):
return status
time.sleep(poll_s)
waited += poll_s
raise TimeoutError(f"Run {run.run_name} still in status {status}")
runs = {}
for feedback_name in FEEDBACKS:
feedback_df = benchmark_df[
benchmark_df["feedback"] == feedback_name
].copy()
# Represent repeated trials as input rows so the Run API parallelizes them.
feedback_df = feedback_df.loc[
feedback_df.index.repeat(N_TRIALS)
].reset_index(drop=True)
feedback_df["trial"] = feedback_df.groupby("input_id").cumcount()
feedback_df["base_input_id"] = feedback_df["input_id"]
feedback_df["input_id"] = (
feedback_df["input_id"] + "-trial-" + feedback_df["trial"].astype(str)
)
run = tru_app.add_run(
run_config=RunConfig(
run_name=f"judge_benchmark_{feedback_name}",
dataset_name=f"judge_benchmark_{feedback_name}",
source_type="DATAFRAME",
dataset_spec={
"input_id": "input_id",
"query": "input",
"answer": "output",
"contexts": "retrieved_contexts",
},
invocation_max_workers=8,
metric_max_workers=4,
)
)
run.start(input_df=feedback_df)
wait_for_invocation(run)
run.compute_metrics([
METRICS[(feedback_name, model)] for model in MODELS
])
runs[feedback_name] = {"run": run, "input_df": feedback_df}
print(f"{feedback_name}: {len(feedback_df)} rows x {len(MODELS)} models")
session.force_flush()
import time
from trulens.core.run import RunConfig
from trulens.core.run import RunStatus
N_TRIALS = 3 # reduce to 1 for a cheap smoke run
def wait_for_invocation(run, timeout_s: int = 600, poll_s: int = 2):
waited = 0
while waited <= timeout_s:
status = run.get_status()
if status in (
RunStatus.INVOCATION_COMPLETED,
RunStatus.INVOCATION_PARTIALLY_COMPLETED,
):
return status
time.sleep(poll_s)
waited += poll_s
raise TimeoutError(f"Run {run.run_name} still in status {status}")
runs = {}
for feedback_name in FEEDBACKS:
feedback_df = benchmark_df[
benchmark_df["feedback"] == feedback_name
].copy()
# Represent repeated trials as input rows so the Run API parallelizes them.
feedback_df = feedback_df.loc[
feedback_df.index.repeat(N_TRIALS)
].reset_index(drop=True)
feedback_df["trial"] = feedback_df.groupby("input_id").cumcount()
feedback_df["base_input_id"] = feedback_df["input_id"]
feedback_df["input_id"] = (
feedback_df["input_id"] + "-trial-" + feedback_df["trial"].astype(str)
)
run = tru_app.add_run(
run_config=RunConfig(
run_name=f"judge_benchmark_{feedback_name}",
dataset_name=f"judge_benchmark_{feedback_name}",
source_type="DATAFRAME",
dataset_spec={
"input_id": "input_id",
"query": "input",
"answer": "output",
"contexts": "retrieved_contexts",
},
invocation_max_workers=8,
metric_max_workers=4,
)
)
run.start(input_df=feedback_df)
wait_for_invocation(run)
run.compute_metrics([
METRICS[(feedback_name, model)] for model in MODELS
])
runs[feedback_name] = {"run": run, "input_df": feedback_df}
print(f"{feedback_name}: {len(feedback_df)} rows x {len(MODELS)} models")
session.force_flush()
Results¶
Accuracy and latency by feedback function¶
run.get_record_details() returns one row per record with a score column per metric. We join it back to the labels on input_id and compute mean absolute error against the human labels with TruLens's GroundTruthAggregator. Judge latency is read from the EVAL_ROOT spans TruLens recorded for each metric computation (the record-level latency column only measures our instant replay app, not the judges).
In [ ]:
Copied!
import json
from trulens.feedback import GroundTruthAggregator
LABEL_COLS = ["input_id", "base_input_id", "domain", "trial", "expected_score"]
def judge_latencies() -> pd.Series:
"""Mean latency per metric, from the EVAL_ROOT spans of each judge call."""
events = session.connector.get_events(
app_name=APP_NAME, app_version=APP_VERSION,
record_ids=None, start_time=None,
)
rows = []
for _, event in events.iterrows():
attrs = event["record_attributes"]
if isinstance(attrs, str):
attrs = json.loads(attrs)
if attrs.get(SpanAttributes.SPAN_TYPE) != SpanAttributes.SpanType.EVAL_ROOT:
continue
rows.append({
"metric_name": attrs.get(SpanAttributes.EVAL_ROOT.METRIC_NAME),
"latency_s": (
event["timestamp"] - event["start_timestamp"]
).total_seconds(),
})
return pd.DataFrame(rows).groupby("metric_name")["latency_s"].mean()
latency_by_metric = judge_latencies()
joined = {}
result_rows = []
for feedback_name, state in runs.items():
details = state["run"].get_record_details()
scored = details.merge(
state["input_df"][LABEL_COLS],
on="input_id",
how="left",
validate="one_to_one",
)
joined[feedback_name] = scored
for model in MODELS:
metric_name = f"{feedback_name}/{model}"
scores = pd.to_numeric(scored[metric_name], errors="coerce")
expected = scored["expected_score"].astype(float)
ok = scores.notna()
agg = GroundTruthAggregator(true_labels=expected[ok].tolist())
result_rows.append({
"feedback": feedback_name,
"model": model,
"mae": agg.mae(scores[ok].tolist()),
"n_scored": int(ok.sum()),
"n_failed": int((~ok).sum()),
"mean_judge_latency_s": latency_by_metric.get(metric_name),
})
summary_df = pd.DataFrame(result_rows).sort_values(["feedback", "mae"])
summary_df
import json
from trulens.feedback import GroundTruthAggregator
LABEL_COLS = ["input_id", "base_input_id", "domain", "trial", "expected_score"]
def judge_latencies() -> pd.Series:
"""Mean latency per metric, from the EVAL_ROOT spans of each judge call."""
events = session.connector.get_events(
app_name=APP_NAME, app_version=APP_VERSION,
record_ids=None, start_time=None,
)
rows = []
for _, event in events.iterrows():
attrs = event["record_attributes"]
if isinstance(attrs, str):
attrs = json.loads(attrs)
if attrs.get(SpanAttributes.SPAN_TYPE) != SpanAttributes.SpanType.EVAL_ROOT:
continue
rows.append({
"metric_name": attrs.get(SpanAttributes.EVAL_ROOT.METRIC_NAME),
"latency_s": (
event["timestamp"] - event["start_timestamp"]
).total_seconds(),
})
return pd.DataFrame(rows).groupby("metric_name")["latency_s"].mean()
latency_by_metric = judge_latencies()
joined = {}
result_rows = []
for feedback_name, state in runs.items():
details = state["run"].get_record_details()
scored = details.merge(
state["input_df"][LABEL_COLS],
on="input_id",
how="left",
validate="one_to_one",
)
joined[feedback_name] = scored
for model in MODELS:
metric_name = f"{feedback_name}/{model}"
scores = pd.to_numeric(scored[metric_name], errors="coerce")
expected = scored["expected_score"].astype(float)
ok = scores.notna()
agg = GroundTruthAggregator(true_labels=expected[ok].tolist())
result_rows.append({
"feedback": feedback_name,
"model": model,
"mae": agg.mae(scores[ok].tolist()),
"n_scored": int(ok.sum()),
"n_failed": int((~ok).sum()),
"mean_judge_latency_s": latency_by_metric.get(metric_name),
})
summary_df = pd.DataFrame(result_rows).sort_values(["feedback", "mae"])
summary_df
Accuracy by use case¶
Mean absolute error per domain, per model, aggregated across the three feedback functions. Because these are tiny illustrative sets, treat close numbers as ties — the point is the workflow, not a verdict on which model is best for an industry in general. Rerun this on labeled examples from your own traces before choosing.
In [ ]:
Copied!
domain_frames = []
for feedback_name, scored in joined.items():
for model in MODELS:
metric_name = f"{feedback_name}/{model}"
scores = pd.to_numeric(scored[metric_name], errors="coerce")
domain_frames.append(
scored.assign(
model=model,
feedback=feedback_name,
abs_error=(
scores - scored["expected_score"].astype(float)
).abs(),
)[["domain", "model", "feedback", "trial", "abs_error"]]
)
domain_errors = pd.concat(domain_frames, ignore_index=True)
domain_summary_df = (
domain_errors.groupby(["domain", "model"])["abs_error"]
.mean()
.unstack()
.round(3)
)
domain_summary_df
domain_frames = []
for feedback_name, scored in joined.items():
for model in MODELS:
metric_name = f"{feedback_name}/{model}"
scores = pd.to_numeric(scored[metric_name], errors="coerce")
domain_frames.append(
scored.assign(
model=model,
feedback=feedback_name,
abs_error=(
scores - scored["expected_score"].astype(float)
).abs(),
)[["domain", "model", "feedback", "trial", "abs_error"]]
)
domain_errors = pd.concat(domain_frames, ignore_index=True)
domain_summary_df = (
domain_errors.groupby(["domain", "model"])["abs_error"]
.mean()
.unstack()
.round(3)
)
domain_summary_df
Winner, ties, and win rate per use case¶
Judge scores are noisy, so a single MAE ranking can overstate how decisive a "winner" is. For each use case we report the lowest-MAE model, which other models are within noise of it (mean MAE within one trial-level standard deviation), and how often the winner actually won across individual trials. If the winner is unstable or several models are within noise, treat them as tied and decide on latency and cost instead.
In [ ]:
Copied!
best_rows = []
for domain, domain_df in domain_errors.groupby("domain"):
per_trial = (
domain_df.groupby(["model", "trial"])["abs_error"]
.mean()
.unstack("trial")
)
means = per_trial.mean(axis=1)
stds = per_trial.std(axis=1).fillna(0.0)
best = means.idxmin()
within_noise = [
model
for model in means.index
if model != best and means[model] <= means[best] + stds[best]
]
win_rate = (per_trial.idxmin(axis=0) == best).mean()
best_rows.append({
"domain": domain,
"best_model": best,
"mae": round(means[best], 3),
"trial_win_rate": round(float(win_rate), 2),
"within_noise_of_best": ", ".join(within_noise) or "-",
})
pd.DataFrame(best_rows)
best_rows = []
for domain, domain_df in domain_errors.groupby("domain"):
per_trial = (
domain_df.groupby(["model", "trial"])["abs_error"]
.mean()
.unstack("trial")
)
means = per_trial.mean(axis=1)
stds = per_trial.std(axis=1).fillna(0.0)
best = means.idxmin()
within_noise = [
model
for model in means.index
if model != best and means[model] <= means[best] + stds[best]
]
win_rate = (per_trial.idxmin(axis=0) == best).mean()
best_rows.append({
"domain": domain,
"best_model": best,
"mae": round(means[best], 3),
"trial_win_rate": round(float(win_rate), 2),
"within_noise_of_best": ", ".join(within_noise) or "-",
})
pd.DataFrame(best_rows)
Observed cost¶
Metric execution tracks endpoint costs automatically (Endpoint.track_all_costs_tally), so get_record_details() includes a "<metric name> feedback cost in USD" column per metric — the observed spend of each judge, not a list-price estimate.
Dollar costs rely on the price table shipped with
langchain_community's OpenAI cost callback. If a model is newer than your installed version's table, its cost shows up as 0 — in that case compare models by token counts or upgradelangchain_community. Cost capture can also vary by feedback implementation: in our runs, groundedness reported per-call cost while the relevance metrics reported 0 for every model, so treat zeros as "not captured", not "free".
In [ ]:
Copied!
cost_rows = []
for feedback_name, scored in joined.items():
for model in MODELS:
metric_name = f"{feedback_name}/{model}"
cost_col = f"{metric_name} feedback cost in USD"
if cost_col in scored:
cost_rows.append({
"feedback": feedback_name,
"model": model,
"total_feedback_cost_usd": scored[cost_col].sum(),
"mean_feedback_cost_usd": scored[cost_col].mean(),
})
cost_df = pd.DataFrame(cost_rows)
if cost_df.empty or cost_df["total_feedback_cost_usd"].sum() == 0:
print(
"No observed USD costs found - your langchain_community version "
"may not have prices for these models. Compare token usage instead."
)
cost_df
cost_rows = []
for feedback_name, scored in joined.items():
for model in MODELS:
metric_name = f"{feedback_name}/{model}"
cost_col = f"{metric_name} feedback cost in USD"
if cost_col in scored:
cost_rows.append({
"feedback": feedback_name,
"model": model,
"total_feedback_cost_usd": scored[cost_col].sum(),
"mean_feedback_cost_usd": scored[cost_col].mean(),
})
cost_df = pd.DataFrame(cost_rows)
if cost_df.empty or cost_df["total_feedback_cost_usd"].sum() == 0:
print(
"No observed USD costs found - your langchain_community version "
"may not have prices for these models. Compare token usage instead."
)
cost_df
Visual comparison¶
In [ ]:
Copied!
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))
domain_summary_df.plot.bar(ax=axes[0], rot=45)
axes[0].set_title("MAE by use case (lower is better)")
axes[0].set_ylabel("Mean absolute error")
summary_df.pivot(
index="feedback", columns="model", values="mean_judge_latency_s"
).plot.bar(ax=axes[1], rot=0)
axes[1].set_title("Mean judge latency per call")
axes[1].set_ylabel("Seconds")
if not cost_df.empty:
cost_df.pivot(
index="feedback", columns="model", values="total_feedback_cost_usd"
).plot.bar(ax=axes[2], rot=0)
axes[2].set_title("Observed judge cost (total USD)")
axes[2].set_ylabel("USD")
plt.tight_layout()
plt.show()
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))
domain_summary_df.plot.bar(ax=axes[0], rot=45)
axes[0].set_title("MAE by use case (lower is better)")
axes[0].set_ylabel("Mean absolute error")
summary_df.pivot(
index="feedback", columns="model", values="mean_judge_latency_s"
).plot.bar(ax=axes[1], rot=0)
axes[1].set_title("Mean judge latency per call")
axes[1].set_ylabel("Seconds")
if not cost_df.empty:
cost_df.pivot(
index="feedback", columns="model", values="total_feedback_cost_usd"
).plot.bar(ax=axes[2], rot=0)
axes[2].set_title("Observed judge cost (total USD)")
axes[2].set_ylabel("USD")
plt.tight_layout()
plt.show()
Choosing your judge¶
Read the recommendation off your summary_df, domain_summary_df, and the win-rate table: pick the lowest-MAE model for the feedback functions and domain that matter to you — provided its trial win rate is stable — then sanity-check its observed latency and cost from the tables above. On toy sets this small, models within noise of each other are effectively tied: decide ties on latency and cost, and rerun with labeled examples from your own traces before committing.
Once you've chosen, set the model everywhere you build a provider:
from trulens.providers.openai import OpenAI
provider = OpenAI(model_engine="gpt-4.1-nano") # your winner here
You can also open the TruLens dashboard (from trulens.dashboard import run_dashboard; run_dashboard(session)) to inspect every judge call, its chain-of-thought reasoning, and its cost, per record.