Evaluating Healthcare Administrative AI with TruLens¶

This notebook demonstrates how to evaluate groundedness in a healthcare administrative conversational AI application. The examples cover claim status, prior authorization, benefits, and next-step guidance.

The notebook evaluates administrative assistance only. It does not evaluate diagnosis, treatment, medical necessity, or other clinical decisions. All members, identifiers, plans, claims, authorizations, and responses below are fictional. Do not use real patient data in this notebook.

In [ ]:

# !pip install trulens trulens-providers-openai openai pandas

# !pip install trulens trulens-providers-openai openai pandas

Configure the evaluation provider¶

Set OPENAI_API_KEY in your environment before running the evaluation. The application responses in this example are deterministic; the API key is used only by the groundedness judge.

In [ ]:

import os

if "OPENAI_API_KEY" not in os.environ:
    raise RuntimeError("Set OPENAI_API_KEY before running this notebook.")

import os if "OPENAI_API_KEY" not in os.environ: raise RuntimeError("Set OPENAI_API_KEY before running this notebook.")

Create synthetic administrative evaluation fixtures¶

Each fixture contains an authoritative administrative record and a candidate response. The expected behavior is included only to make the test set reviewable; it is not passed to the evaluator.

The four cases exercise:

1. a supported claim-status response;
2. an unsupported prior-authorization assertion;
3. a benefit explanation that contradicts its source; and
4. an appropriate abstention when a next-step deadline is unavailable.

In [ ]:

SYNTHETIC_CASES = [
    {
        "scenario": "claim_status",
        "expected_behavior": "supported",
        "query": "What is the status of synthetic claim CLM-1001?",
        "authoritative_context": (
            "Synthetic claim CLM-1001 was received on May 2, 2026. "
            "It was processed and paid on May 10, 2026. The plan paid "
            "$120.00, and the synthetic member responsibility is $30.00."
        ),
        "candidate_response": (
            "Claim CLM-1001 was processed and paid on May 10, 2026. "
            "The plan paid $120.00, and your responsibility is $30.00."
        ),
    },
    {
        "scenario": "prior_authorization",
        "expected_behavior": "unsupported",
        "query": "Has synthetic authorization PA-204 been approved?",
        "authoritative_context": (
            "Synthetic prior authorization PA-204 is pending. The request "
            "is waiting for additional documentation. No approval decision "
            "or authorization period is recorded."
        ),
        "candidate_response": (
            "Prior authorization PA-204 is approved through December 31, 2026."
        ),
    },
    {
        "scenario": "benefit_explanation",
        "expected_behavior": "contradictory",
        "query": "What is the synthetic plan's urgent care benefit?",
        "authoritative_context": (
            "For the synthetic Silver Plan, in-network urgent care has a "
            "$50 copayment after the annual deductible. The annual "
            "deductible is $1,500, of which $600 has been met. This is a "
            "benefit summary, not a guarantee of payment."
        ),
        "candidate_response": (
            "In-network urgent care has a $20 copayment, and the deductible "
            "does not apply. Payment is guaranteed."
        ),
    },
    {
        "scenario": "next_step_guidance",
        "expected_behavior": "appropriate_abstention",
        "query": (
            "What is the filing deadline for an appeal of synthetic claim "
            "CLM-1002?"
        ),
        "authoritative_context": (
            "Synthetic claim CLM-1002 was denied. The available record does "
            "not include an appeal filing deadline or appeal instructions."
        ),
        "candidate_response": (
            "The available record does not include the appeal deadline, so "
            "I cannot confirm one from the information provided."
        ),
    },
]

SYNTHETIC_CASES = [ { "scenario": "claim_status", "expected_behavior": "supported", "query": "What is the status of synthetic claim CLM-1001?", "authoritative_context": ( "Synthetic claim CLM-1001 was received on May 2, 2026. " "It was processed and paid on May 10, 2026. The plan paid " "$120.00, and the synthetic member responsibility is $30.00." ), "candidate_response": ( "Claim CLM-1001 was processed and paid on May 10, 2026. " "The plan paid $120.00, and your responsibility is $30.00." ), }, { "scenario": "prior_authorization", "expected_behavior": "unsupported", "query": "Has synthetic authorization PA-204 been approved?", "authoritative_context": ( "Synthetic prior authorization PA-204 is pending. The request " "is waiting for additional documentation. No approval decision " "or authorization period is recorded." ), "candidate_response": ( "Prior authorization PA-204 is approved through December 31, 2026." ), }, { "scenario": "benefit_explanation", "expected_behavior": "contradictory", "query": "What is the synthetic plan's urgent care benefit?", "authoritative_context": ( "For the synthetic Silver Plan, in-network urgent care has a " "$50 copayment after the annual deductible. The annual " "deductible is $1,500, of which $600 has been met. This is a " "benefit summary, not a guarantee of payment." ), "candidate_response": ( "In-network urgent care has a $20 copayment, and the deductible " "does not apply. Payment is guaranteed." ), }, { "scenario": "next_step_guidance", "expected_behavior": "appropriate_abstention", "query": ( "What is the filing deadline for an appeal of synthetic claim " "CLM-1002?" ), "authoritative_context": ( "Synthetic claim CLM-1002 was denied. The available record does " "not include an appeal filing deadline or appeal instructions." ), "candidate_response": ( "The available record does not include the appeal deadline, so " "I cannot confirm one from the information provided." ), }, ]

In [ ]:

import pandas as pd

fixtures = pd.DataFrame(SYNTHETIC_CASES)
fixtures[["scenario", "expected_behavior", "query"]]

import pandas as pd fixtures = pd.DataFrame(SYNTHETIC_CASES) fixtures[["scenario", "expected_behavior", "query"]]

Define a deterministic administrative application¶

The example application retrieves the authoritative context associated with each synthetic query and returns the fixture's candidate response. Instrumentation records the query, retrieved context, and response so Selector can route them to the metric.

In [ ]:

from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes


class HealthcareAdminFixtureApp:
    """Return candidate responses from synthetic evaluation fixtures."""

    def __init__(self, cases: list[dict]):
        self.cases_by_query = {case["query"]: case for case in cases}

    @instrument(
        span_type=SpanAttributes.SpanType.RETRIEVAL,
        attributes={
            SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
            SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
        },
    )
    def retrieve(self, query: str) -> list[str]:
        return [self.cases_by_query[query]["authoritative_context"]]

    @instrument(
        span_type=SpanAttributes.SpanType.RECORD_ROOT,
        attributes={
            SpanAttributes.RECORD_ROOT.INPUT: "query",
            SpanAttributes.RECORD_ROOT.OUTPUT: "return",
        },
    )
    def respond(self, query: str) -> str:
        self.retrieve(query=query)
        return self.cases_by_query[query]["candidate_response"]

from trulens.core.otel.instrument import instrument from trulens.otel.semconv.trace import SpanAttributes class HealthcareAdminFixtureApp: """Return candidate responses from synthetic evaluation fixtures.""" def __init__(self, cases: list[dict]): self.cases_by_query = {case["query"]: case for case in cases} @instrument( span_type=SpanAttributes.SpanType.RETRIEVAL, attributes={ SpanAttributes.RETRIEVAL.QUERY_TEXT: "query", SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return", }, ) def retrieve(self, query: str) -> list[str]: return [self.cases_by_query[query]["authoritative_context"]] @instrument( span_type=SpanAttributes.SpanType.RECORD_ROOT, attributes={ SpanAttributes.RECORD_ROOT.INPUT: "query", SpanAttributes.RECORD_ROOT.OUTPUT: "return", }, ) def respond(self, query: str) -> str: self.retrieve(query=query) return self.cases_by_query[query]["candidate_response"]

Configure Healthcare Groundedness¶

Healthcare Groundedness reuses TruLens's stock groundedness implementation. Domain-specific criteria make the judge strict about administrative facts such as status, dates, dollar amounts, identifiers, coverage conditions, deadlines, and guarantees of payment.

An appropriate statement of uncertainty is grounded when the source does not contain the requested fact. This rewards abstention instead of encouraging the assistant to invent a deadline or status.

In [ ]:

from trulens.core import Metric
from trulens.core import Selector
from trulens.providers.openai import OpenAI

HEALTHCARE_ADMIN_GROUNDEDNESS_CRITERIA = """
Evaluate whether every substantive administrative statement is supported by
the authoritative source. Be strict about claim and authorization status,
dates, dollar amounts, identifiers, coverage conditions, deadlines, required
documents, next steps, and guarantees of payment. A contradiction or invented
administrative fact should receive the lowest score. An explicit statement
that the available source does not contain a requested fact is grounded when
the source genuinely omits that fact.
"""

HEALTHCARE_ADMIN_INSTRUCTIONS = """
Evaluate administrative accuracy only. Do not infer diagnosis, treatment,
medical necessity, or facts from general model knowledge. Use only the supplied
source. Treat an authorization as distinct from a guarantee of claim payment.
"""

provider = OpenAI(model_engine="gpt-4o-mini")

healthcare_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons,
    name="Healthcare Groundedness",
    description=(
        "Measures whether a healthcare administrative response is supported "
        "by the supplied authoritative context."
    ),
    criteria=HEALTHCARE_ADMIN_GROUNDEDNESS_CRITERIA,
    additional_instructions=HEALTHCARE_ADMIN_INSTRUCTIONS,
    selectors={
        "source": Selector.select_context(collect_list=True),
        "statement": Selector.select_record_output(),
    },
)

from trulens.core import Metric from trulens.core import Selector from trulens.providers.openai import OpenAI HEALTHCARE_ADMIN_GROUNDEDNESS_CRITERIA = """ Evaluate whether every substantive administrative statement is supported by the authoritative source. Be strict about claim and authorization status, dates, dollar amounts, identifiers, coverage conditions, deadlines, required documents, next steps, and guarantees of payment. A contradiction or invented administrative fact should receive the lowest score. An explicit statement that the available source does not contain a requested fact is grounded when the source genuinely omits that fact. """ HEALTHCARE_ADMIN_INSTRUCTIONS = """ Evaluate administrative accuracy only. Do not infer diagnosis, treatment, medical necessity, or facts from general model knowledge. Use only the supplied source. Treat an authorization as distinct from a guarantee of claim payment. """ provider = OpenAI(model_engine="gpt-4o-mini") healthcare_groundedness = Metric( implementation=provider.groundedness_measure_with_cot_reasons, name="Healthcare Groundedness", description=( "Measures whether a healthcare administrative response is supported " "by the supplied authoritative context." ), criteria=HEALTHCARE_ADMIN_GROUNDEDNESS_CRITERIA, additional_instructions=HEALTHCARE_ADMIN_INSTRUCTIONS, selectors={ "source": Selector.select_context(collect_list=True), "statement": Selector.select_record_output(), }, )

Record and evaluate the fixtures¶

Use an isolated SQLite database for this example. RunConfig maps the query column to the application's input, and compute_metrics evaluates every recorded response.

In [ ]:

from trulens.apps.app import TruApp
from trulens.core import TruSession
from trulens.core.run import RunConfig

session = TruSession(
    database_url="sqlite:///healthcare_admin_evaluation.sqlite"
)
fixture_app = HealthcareAdminFixtureApp(SYNTHETIC_CASES)

tru_fixture_app = TruApp(
    fixture_app,
    app_name="Healthcare Administrative Evaluation Fixtures",
    app_version="groundedness-v1",
    main_method=fixture_app.respond,
    feedbacks=[healthcare_groundedness],
)

evaluation_run = tru_fixture_app.add_run(
    run_config=RunConfig(
        run_name="synthetic_healthcare_admin_groundedness",
        dataset_name="synthetic_healthcare_admin_cases",
        source_type="DATAFRAME",
        dataset_spec={"input": "query"},
    )
)

from trulens.apps.app import TruApp from trulens.core import TruSession from trulens.core.run import RunConfig session = TruSession( database_url="sqlite:///healthcare_admin_evaluation.sqlite" ) fixture_app = HealthcareAdminFixtureApp(SYNTHETIC_CASES) tru_fixture_app = TruApp( fixture_app, app_name="Healthcare Administrative Evaluation Fixtures", app_version="groundedness-v1", main_method=fixture_app.respond, feedbacks=[healthcare_groundedness], ) evaluation_run = tru_fixture_app.add_run( run_config=RunConfig( run_name="synthetic_healthcare_admin_groundedness", dataset_name="synthetic_healthcare_admin_cases", source_type="DATAFRAME", dataset_spec={"input": "query"}, ) )

In [ ]:

evaluation_run.start(input_df=fixtures[["query"]])

evaluation_run.start(input_df=fixtures[["query"]])

In [ ]:

evaluation_run.compute_metrics([healthcare_groundedness])

evaluation_run.compute_metrics([healthcare_groundedness])

Review the results¶

The supported claim response and appropriate abstention should score above the invented authorization approval and contradictory benefit explanation. The check below verifies that ordering without treating a particular numeric threshold as a compliance decision. Always inspect the reasons and supporting evidence because judge scores can vary by model.

In [ ]:

records = evaluation_run.get_records()
results = records.merge(
    fixtures[["query", "scenario", "expected_behavior"]],
    left_on="input",
    right_on="query",
)
results[
    [
        "scenario",
        "expected_behavior",
        "Healthcare Groundedness",
    ]
]

records = evaluation_run.get_records() results = records.merge( fixtures[["query", "scenario", "expected_behavior"]], left_on="input", right_on="query", ) results[ [ "scenario", "expected_behavior", "Healthcare Groundedness", ] ]

In [ ]:

expected_high = results[
    results["expected_behavior"].isin(["supported", "appropriate_abstention"])
]["Healthcare Groundedness"]
expected_low = results[
    results["expected_behavior"].isin(["unsupported", "contradictory"])
]["Healthcare Groundedness"]

assert expected_high.min() > expected_low.max(), (
    "Expected supported and abstaining responses to score above "
    "unsupported and contradictory responses. Review the judge reasons."
)

expected_high = results[ results["expected_behavior"].isin(["supported", "appropriate_abstention"]) ]["Healthcare Groundedness"] expected_low = results[ results["expected_behavior"].isin(["unsupported", "contradictory"]) ]["Healthcare Groundedness"] assert expected_high.min() > expected_low.max(), ( "Expected supported and abstaining responses to score above " "unsupported and contradictory responses. Review the judge reasons." )

In [ ]:

session.get_leaderboard()

session.get_leaderboard()

Safety and interpretation notes¶

• All data in this example is synthetic and intentionally minimal.
• Do not place real protected health information in example notebooks, prompts, traces, or evaluation reasons.
• Groundedness measures support from the supplied source; it does not prove that the source itself is correct or current.
• This example is not a HIPAA, privacy, legal, clinical-safety, or regulatory compliance assessment.
• Production evaluations should use organization-approved data handling, access control, retention, human review, and model governance processes.