Conversation Evaluation Quickstart¶

This quickstart uses OpenAI to record a multi-turn conversation in one conversation_id context, evaluates it once with TruLens' built-in Conversation Helpfulness metric, waits for the persisted result, and opens the conversation-aware dashboard.

Import dependencies¶

Load OpenAI and the TruLens APIs used to define, instrument, record, and evaluate the application.

In [ ]:

from openai import OpenAI
from trulens.apps.app import TruApp
from trulens.core import Metric
from trulens.core.database.connector.default import DefaultDBConnector
from trulens.core.otel.instrument import instrument
from trulens.core.session import TruSession
from trulens.providers.openai import OpenAI as OpenAIProvider

from openai import OpenAI from trulens.apps.app import TruApp from trulens.core import Metric from trulens.core.database.connector.default import DefaultDBConnector from trulens.core.otel.instrument import instrument from trulens.core.session import TruSession from trulens.providers.openai import OpenAI as OpenAIProvider

Configure the TruLens database¶

Create an explicit SQLite-backed connector and pass it to TruSession.

In [ ]:

connector = DefaultDBConnector(
    database_url="sqlite:///conversation_metrics_demo.sqlite"
)
session = TruSession(connector=connector)
session.reset_database()

connector = DefaultDBConnector( database_url="sqlite:///conversation_metrics_demo.sqlite" ) session = TruSession(connector=connector) session.reset_database()

Define the support assistant¶

Create a stateful OpenAI-backed assistant. The instrumented respond method becomes the recorded application call for each turn.

In [ ]:

class SupportAssistant:
    def __init__(self, model: str = "gpt-4.1-mini"):
        self.client = OpenAI()
        self.model = model
        self.messages = [
            {
                "role": "system",
                "content": (
                    "You are a concise customer support assistant. "
                    "Answer each question using the preceding conversation for context."
                ),
            }
        ]

    def reset(self) -> None:
        self.messages = self.messages[:1]

    @instrument()
    def respond(self, question: str) -> str:
        self.messages.append({"role": "user", "content": question})
        completion = self.client.chat.completions.create(
            model=self.model,
            temperature=0,
            messages=self.messages,
        )
        response = completion.choices[0].message.content or ""
        self.messages.append({"role": "assistant", "content": response})
        return response

class SupportAssistant: def __init__(self, model: str = "gpt-4.1-mini"): self.client = OpenAI() self.model = model self.messages = [ { "role": "system", "content": ( "You are a concise customer support assistant. " "Answer each question using the preceding conversation for context." ), } ] def reset(self) -> None: self.messages = self.messages[:1] @instrument() def respond(self, question: str) -> str: self.messages.append({"role": "user", "content": question}) completion = self.client.chat.completions.create( model=self.model, temperature=0, messages=self.messages, ) response = completion.choices[0].message.content or "" self.messages.append({"role": "assistant", "content": response}) return response

Define the conversation metric¶

Use TruLens' built-in conversation-helpfulness evaluator to score the assistant across the complete ordered conversation.

In [ ]:

provider = OpenAIProvider(model_engine="gpt-4.1-mini")

metric = Metric(
    implementation=provider.conversation_helpfulness,
    name="Conversation Helpfulness",
).on_conversation()

provider = OpenAIProvider(model_engine="gpt-4.1-mini") metric = Metric( implementation=provider.conversation_helpfulness, name="Conversation Helpfulness", ).on_conversation()

Configure the recorder¶

Wrap the assistant with TruApp and attach the conversation-level metric.

In [ ]:

app = SupportAssistant()
recorder = TruApp(
    app,
    app_name="Conversation Metrics Demo",
    app_version="gpt-4.1-mini",
    main_method=app.respond,
    feedbacks=[metric],
)

app = SupportAssistant() recorder = TruApp( app, app_name="Conversation Metrics Demo", app_version="gpt-4.1-mini", main_method=app.respond, feedbacks=[metric], )

Record conversations¶

Record a 12-turn password-reset conversation and a separate invoice-export conversation by reusing a conversation_id across turns.

In [ ]:

password_reset_questions = [
    "How do I reset my password?",
    "Where is Security?",
    "I do not see Reset password.",
    "Will that sign me out everywhere?",
    "Can I keep this browser signed in?",
    "What if I cannot access my email?",
    "Can an admin reset it for me?",
    "How long is the temporary password valid?",
    "Do I need MFA after the reset?",
    "Can I change my MFA device too?",
    "Is the password history enforced?",
    "Can you summarize the steps?",
]

with recorder(conversation_id="password-reset") as password_reset_recording:
    for question in password_reset_questions:
        app.respond(question)

app.reset()
with recorder(conversation_id="invoice-export") as invoice_export_recording:
    app.respond("Can I export invoices?")

password_reset_questions = [ "How do I reset my password?", "Where is Security?", "I do not see Reset password.", "Will that sign me out everywhere?", "Can I keep this browser signed in?", "What if I cannot access my email?", "Can an admin reset it for me?", "How long is the temporary password valid?", "Do I need MFA after the reset?", "Can I change my MFA device too?", "Is the password history enforced?", "Can you summarize the steps?", ] with recorder(conversation_id="password-reset") as password_reset_recording: for question in password_reset_questions: app.respond(question) app.reset() with recorder(conversation_id="invoice-export") as invoice_export_recording: app.respond("Can I export invoices?")

Wait for conversation feedback¶

Conversation evaluation is queued automatically when each recording context exits. Wait for both exact batches before opening the dashboard.

In [ ]:

password_reset_recording.retrieve_feedback_results()
invoice_export_recording.retrieve_feedback_results()

password_reset_recording.retrieve_feedback_results() invoice_export_recording.retrieve_feedback_results()

Dashboard¶

Run the next cell to open the dashboard backed by this notebook's SQLite database. Stop it with session.stop_dashboard().

In [ ]:

from trulens.dashboard import run_dashboard

dashboard = run_dashboard(session=session, force=True)
dashboard

from trulens.dashboard import run_dashboard dashboard = run_dashboard(session=session, force=True) dashboard