Batch Evaluation with TruLens Runs¶

This quickstart shows how to run batch evaluations using TruLens Run APIs.

Instead of manually looping over queries in a recording context, the Run API gives you a structured workflow:

1. Define a RunConfig pointing to your test dataset
2. Call app.add_run(run_config) to create a run
3. Call run.start(input_df=df) to invoke your app on every row
4. Call run.compute_metrics([...]) to evaluate the traces
5. Call run.get_records() to inspect results

We build a simple RAG over national parks data, define the hallucination triad as metrics, and compare two model versions.

In [ ]:

# !pip install trulens trulens-providers-openai chromadb openai

# !pip install trulens trulens-providers-openai chromadb openai

In [ ]:

import os

if "OPENAI_API_KEY" not in os.environ:
    os.environ["OPENAI_API_KEY"] = "sk-proj-..."

import os if "OPENAI_API_KEY" not in os.environ: os.environ["OPENAI_API_KEY"] = "sk-proj-..."

Set up knowledge base¶

We create a small knowledge base about national parks and populate a vector store.

In [ ]:

yellowstone_info = """
Yellowstone National Park, established in 1872, was the world's first national park.
Located primarily in Wyoming, it spans nearly 3,500 square miles and sits atop a volcanic hotspot.
The park is famous for its geothermal features, including Old Faithful geyser, and is home to
grizzly bears, wolves, bison, and elk.
"""

yosemite_info = """
Yosemite National Park is located in California's Sierra Nevada mountains.
It is known for its granite cliffs, waterfalls, giant sequoia groves, and biological diversity.
El Capitan and Half Dome are among the most iconic rock formations in the world.
The park covers about 750,000 acres and was instrumental in the development of the national park idea.
"""

grand_canyon_info = """
The Grand Canyon, carved by the Colorado River over millions of years, is one of the most
spectacular geological features on Earth. Located in Arizona, the canyon is 277 miles long,
up to 18 miles wide, and over a mile deep. Grand Canyon National Park was established in 1919.
"""

zion_info = """
Zion National Park is located in southwestern Utah and is known for its steep red cliffs.
The park's main feature is Zion Canyon, which is 15 miles long and up to 2,640 feet deep.
The Virgin River runs through the canyon. Popular hikes include Angels Landing and The Narrows.
"""

glacier_info = """
Glacier National Park in Montana contains over 700 miles of hiking trails, numerous glacially
carved lakes, and the famous Going-to-the-Sun Road. The park is part of the Crown of the
Continent ecosystem and is home to grizzly bears, mountain goats, and wolverines.
"""

acadia_info = """
Acadia National Park, located on Mount Desert Island in Maine, protects the natural beauty
of the highest rocky headlands along the Atlantic coastline. The park includes woodlands,
rocky beaches, and glacier-scoured granite peaks such as Cadillac Mountain.
"""

yellowstone_info = """ Yellowstone National Park, established in 1872, was the world's first national park. Located primarily in Wyoming, it spans nearly 3,500 square miles and sits atop a volcanic hotspot. The park is famous for its geothermal features, including Old Faithful geyser, and is home to grizzly bears, wolves, bison, and elk. """ yosemite_info = """ Yosemite National Park is located in California's Sierra Nevada mountains. It is known for its granite cliffs, waterfalls, giant sequoia groves, and biological diversity. El Capitan and Half Dome are among the most iconic rock formations in the world. The park covers about 750,000 acres and was instrumental in the development of the national park idea. """ grand_canyon_info = """ The Grand Canyon, carved by the Colorado River over millions of years, is one of the most spectacular geological features on Earth. Located in Arizona, the canyon is 277 miles long, up to 18 miles wide, and over a mile deep. Grand Canyon National Park was established in 1919. """ zion_info = """ Zion National Park is located in southwestern Utah and is known for its steep red cliffs. The park's main feature is Zion Canyon, which is 15 miles long and up to 2,640 feet deep. The Virgin River runs through the canyon. Popular hikes include Angels Landing and The Narrows. """ glacier_info = """ Glacier National Park in Montana contains over 700 miles of hiking trails, numerous glacially carved lakes, and the famous Going-to-the-Sun Road. The park is part of the Crown of the Continent ecosystem and is home to grizzly bears, mountain goats, and wolverines. """ acadia_info = """ Acadia National Park, located on Mount Desert Island in Maine, protects the natural beauty of the highest rocky headlands along the Atlantic coastline. The park includes woodlands, rocky beaches, and glacier-scoured granite peaks such as Cadillac Mountain. """

In [ ]:

import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

embedding_function = OpenAIEmbeddingFunction(
    api_key=os.environ.get("OPENAI_API_KEY"),
    model_name="text-embedding-3-small",
)

chroma_client = chromadb.Client()
vector_store = chroma_client.get_or_create_collection(
    name="national_parks", embedding_function=embedding_function
)

vector_store.add("yellowstone", documents=yellowstone_info)
vector_store.add("yosemite", documents=yosemite_info)
vector_store.add("grand_canyon", documents=grand_canyon_info)
vector_store.add("zion", documents=zion_info)
vector_store.add("glacier", documents=glacier_info)
vector_store.add("acadia", documents=acadia_info)

import chromadb from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction embedding_function = OpenAIEmbeddingFunction( api_key=os.environ.get("OPENAI_API_KEY"), model_name="text-embedding-3-small", ) chroma_client = chromadb.Client() vector_store = chroma_client.get_or_create_collection( name="national_parks", embedding_function=embedding_function ) vector_store.add("yellowstone", documents=yellowstone_info) vector_store.add("yosemite", documents=yosemite_info) vector_store.add("grand_canyon", documents=grand_canyon_info) vector_store.add("zion", documents=zion_info) vector_store.add("glacier", documents=glacier_info) vector_store.add("acadia", documents=acadia_info)

Build the RAG application¶

We define a simple RAG with TruLens OTEL instrumentation. The @instrument decorator tags each method with a span type so that Selector can find the right data at evaluation time.

In [ ]:

from openai import OpenAI
from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes

oai_client = OpenAI()


class RAG:
    def __init__(self, model_name: str = "gpt-4o-mini"):
        self.model_name = model_name

    @instrument(
        span_type=SpanAttributes.SpanType.RETRIEVAL,
        attributes={
            SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
            SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
        },
    )
    def retrieve(self, query: str) -> list:
        results = vector_store.query(query_texts=query, n_results=3)
        return [doc for sublist in results["documents"] for doc in sublist]

    @instrument(span_type=SpanAttributes.SpanType.GENERATION)
    def generate_completion(self, query: str, context_list: list) -> str:
        if not context_list:
            return "I don't have enough information to answer this question."

        context = "\n---\n".join(context_list)

        completion = (
            oai_client.chat.completions.create(
                model=self.model_name,
                messages=[
                    {
                        "role": "system",
                        "content": "You are a national parks expert. Answer questions using only the provided context.",
                    },
                    {
                        "role": "user",
                        "content": f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer:",
                    },
                ],
            )
            .choices[0]
            .message.content
        )
        return completion or "No answer generated."

    @instrument(
        span_type=SpanAttributes.SpanType.RECORD_ROOT,
        attributes={
            SpanAttributes.RECORD_ROOT.INPUT: "query",
            SpanAttributes.RECORD_ROOT.OUTPUT: "return",
        },
    )
    def query(self, query: str) -> str:
        context_list = self.retrieve(query=query)
        completion = self.generate_completion(
            query=query, context_list=context_list
        )
        return completion

from openai import OpenAI from trulens.core.otel.instrument import instrument from trulens.otel.semconv.trace import SpanAttributes oai_client = OpenAI() class RAG: def __init__(self, model_name: str = "gpt-4o-mini"): self.model_name = model_name @instrument( span_type=SpanAttributes.SpanType.RETRIEVAL, attributes={ SpanAttributes.RETRIEVAL.QUERY_TEXT: "query", SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return", }, ) def retrieve(self, query: str) -> list: results = vector_store.query(query_texts=query, n_results=3) return [doc for sublist in results["documents"] for doc in sublist] @instrument(span_type=SpanAttributes.SpanType.GENERATION) def generate_completion(self, query: str, context_list: list) -> str: if not context_list: return "I don't have enough information to answer this question." context = "\n---\n".join(context_list) completion = ( oai_client.chat.completions.create( model=self.model_name, messages=[ { "role": "system", "content": "You are a national parks expert. Answer questions using only the provided context.", }, { "role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer:", }, ], ) .choices[0] .message.content ) return completion or "No answer generated." @instrument( span_type=SpanAttributes.SpanType.RECORD_ROOT, attributes={ SpanAttributes.RECORD_ROOT.INPUT: "query", SpanAttributes.RECORD_ROOT.OUTPUT: "return", }, ) def query(self, query: str) -> str: context_list = self.retrieve(query=query) completion = self.generate_completion( query=query, context_list=context_list ) return completion

Define the test dataset¶

run.start() accepts a pandas DataFrame. The column names must match the dataset_spec you provide in the RunConfig.

In [ ]:

import pandas as pd

test_dataset = pd.DataFrame({
    "input": [
        "When was Yellowstone established and where is it located?",
        "What are the famous rock formations in Yosemite?",
        "How deep is the Grand Canyon?",
        "What river runs through Zion Canyon?",
        "What wildlife can be found in Glacier National Park?",
        "Where is Acadia National Park located?",
        "Which national park has geothermal features?",
        "What is the Going-to-the-Sun Road?",
    ]
})
test_dataset

import pandas as pd test_dataset = pd.DataFrame({ "input": [ "When was Yellowstone established and where is it located?", "What are the famous rock formations in Yosemite?", "How deep is the Grand Canyon?", "What river runs through Zion Canyon?", "What wildlife can be found in Glacier National Park?", "Where is Acadia National Park located?", "Which national park has geothermal features?", "What is the Going-to-the-Sun Road?", ] }) test_dataset

Define metrics¶

We use the hallucination triad: groundedness, answer relevance, and context relevance. These are Metric objects that will be passed to run.compute_metrics() later.

In [ ]:

import numpy as np
from trulens.core import Metric
from trulens.core import Selector
from trulens.providers.openai import OpenAI as fOpenAI

provider = fOpenAI(model_engine="gpt-4o-mini")

f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons_consider_answerability,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(collect_list=True),
        "statement": Selector.select_record_output(),
        "question": Selector.select_record_input(),
    },
)

f_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=False),
    },
    agg=np.mean,
)

import numpy as np from trulens.core import Metric from trulens.core import Selector from trulens.providers.openai import OpenAI as fOpenAI provider = fOpenAI(model_engine="gpt-4o-mini") f_groundedness = Metric( implementation=provider.groundedness_measure_with_cot_reasons_consider_answerability, name="Groundedness", selectors={ "source": Selector.select_context(collect_list=True), "statement": Selector.select_record_output(), "question": Selector.select_record_input(), }, ) f_answer_relevance = Metric( implementation=provider.relevance_with_cot_reasons, name="Answer Relevance", selectors={ "prompt": Selector.select_record_input(), "response": Selector.select_record_output(), }, ) f_context_relevance = Metric( implementation=provider.context_relevance_with_cot_reasons, name="Context Relevance", selectors={ "question": Selector.select_record_input(), "context": Selector.select_context(collect_list=False), }, agg=np.mean, )

Initialize TruSession¶

Start a fresh session so results are clean.

In [ ]:

from trulens.core import TruSession

session = TruSession()
session.reset_database()

from trulens.core import TruSession session = TruSession() session.reset_database()

Batch evaluation -- Version 1¶

The batch evaluation workflow:

1. Wrap the RAG with TruApp, passing main_method so the Run knows which method to invoke per row.
2. Create a RunConfig that maps the DataFrame column to the app's input.
3. run.start(input_df=...) invokes the app on every row and captures OTEL traces.
4. run.compute_metrics(...) evaluates the captured traces with the hallucination triad.

In [ ]:

from trulens.apps.app import TruApp
from trulens.core.run import RunConfig

rag_v1 = RAG(model_name="gpt-4o-mini")

tru_rag_v1 = TruApp(
    rag_v1,
    app_name="National Parks RAG",
    app_version="v1-gpt-4o-mini",
    main_method=rag_v1.query,
    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
)

run_v1 = tru_rag_v1.add_run(
    run_config=RunConfig(
        run_name="v1_eval",
        dataset_name="test_questions",
        source_type="DATAFRAME",
        dataset_spec={"input": "input"},
    )
)

from trulens.apps.app import TruApp from trulens.core.run import RunConfig rag_v1 = RAG(model_name="gpt-4o-mini") tru_rag_v1 = TruApp( rag_v1, app_name="National Parks RAG", app_version="v1-gpt-4o-mini", main_method=rag_v1.query, feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance], ) run_v1 = tru_rag_v1.add_run( run_config=RunConfig( run_name="v1_eval", dataset_name="test_questions", source_type="DATAFRAME", dataset_spec={"input": "input"}, ) )

In [ ]:

run_v1.start(input_df=test_dataset)

run_v1.start(input_df=test_dataset)

In [ ]:

run_v1.compute_metrics([
    f_groundedness,
    f_answer_relevance,
    f_context_relevance,
])

run_v1.compute_metrics([ f_groundedness, f_answer_relevance, f_context_relevance, ])

In [ ]:

run_v1.get_records()

run_v1.get_records()

Batch evaluation -- Version 2¶

Same dataset, different model. Run the identical workflow with gpt-4o so we can compare.

In [ ]:

rag_v2 = RAG(model_name="gpt-4o")

tru_rag_v2 = TruApp(
    rag_v2,
    app_name="National Parks RAG",
    app_version="v2-gpt-4o",
    main_method=rag_v2.query,
    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
)

run_v2 = tru_rag_v2.add_run(
    run_config=RunConfig(
        run_name="v2_eval",
        dataset_name="test_questions",
        source_type="DATAFRAME",
        dataset_spec={"input": "input"},
    )
)

run_v2.start(input_df=test_dataset)

rag_v2 = RAG(model_name="gpt-4o") tru_rag_v2 = TruApp( rag_v2, app_name="National Parks RAG", app_version="v2-gpt-4o", main_method=rag_v2.query, feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance], ) run_v2 = tru_rag_v2.add_run( run_config=RunConfig( run_name="v2_eval", dataset_name="test_questions", source_type="DATAFRAME", dataset_spec={"input": "input"}, ) ) run_v2.start(input_df=test_dataset)

In [ ]:

run_v2.compute_metrics([
    f_groundedness,
    f_answer_relevance,
    f_context_relevance,
])

run_v2.compute_metrics([ f_groundedness, f_answer_relevance, f_context_relevance, ])

In [ ]:

run_v2.get_records()

run_v2.get_records()

Compare results¶

The leaderboard shows aggregate metrics for each app version side by side.

In [ ]:

session.get_leaderboard()

session.get_leaderboard()

Launch the dashboard¶

Explore individual records, drill into per-query metric scores, and compare versions.

In [ ]:

from trulens.dashboard import run_dashboard

run_dashboard(session)

from trulens.dashboard import run_dashboard run_dashboard(session)