LangChain Conversation Evaluation¶

This framework example shows how to evaluate multi-turn LangChain conversations with TruLens:

1. Build a memory-enabled LangChain chatbot
2. Record two 3-turn conversations, each tagged with a conversation_id
3. Run Answer Relevance for each turn
4. Run Coherence Across Turns once for each ordered conversation
5. Compare turn-level and conversation-level results
6. Explore conversations in the TruLens dashboard

Install dependencies¶

Install TruLens, its LangChain and OpenAI integrations, and the LangChain OpenAI packages when running outside this repository.

In [ ]:

# !pip install trulens trulens-apps-langchain trulens-providers-openai langchain langchain-openai

# !pip install trulens trulens-apps-langchain trulens-providers-openai langchain langchain-openai

Configure OpenAI¶

Provide the API key used by both the chatbot and the OpenAI evaluation provider.

In [ ]:

import os

if "OPENAI_API_KEY" not in os.environ:
    os.environ["OPENAI_API_KEY"] = "sk-proj-..."

import os if "OPENAI_API_KEY" not in os.environ: os.environ["OPENAI_API_KEY"] = "sk-proj-..."

Build the chatbot¶

We use LangChain's RunnableWithMessageHistory to maintain per-conversation history. Each conversation is keyed by a session ID — the same value we will pass as conversation_id to TruLens.

In [ ]:

from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.prompts import MessagesPlaceholder
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

prompt = ChatPromptTemplate.from_messages([
    (
        "system",
        "You are a knowledgeable assistant. Answer concisely in 2–3 sentences.",
    ),
    MessagesPlaceholder(variable_name="history"),
    ("human", "{input}"),
])

history_store: dict[str, InMemoryChatMessageHistory] = {}


def get_session_history(session_id: str) -> InMemoryChatMessageHistory:
    if session_id not in history_store:
        history_store[session_id] = InMemoryChatMessageHistory()
    return history_store[session_id]


chatbot = RunnableWithMessageHistory(
    prompt | llm | StrOutputParser(),
    get_session_history,
    input_messages_key="input",
    history_messages_key="history",
)

from langchain_core.chat_history import InMemoryChatMessageHistory from langchain_core.output_parsers import StrOutputParser from langchain_core.prompts import ChatPromptTemplate from langchain_core.prompts import MessagesPlaceholder from langchain_core.runnables.history import RunnableWithMessageHistory from langchain_openai import ChatOpenAI llm = ChatOpenAI(model="gpt-4o-mini", temperature=0) prompt = ChatPromptTemplate.from_messages([ ( "system", "You are a knowledgeable assistant. Answer concisely in 2–3 sentences.", ), MessagesPlaceholder(variable_name="history"), ("human", "{input}"), ]) history_store: dict[str, InMemoryChatMessageHistory] = {} def get_session_history(session_id: str) -> InMemoryChatMessageHistory: if session_id not in history_store: history_store[session_id] = InMemoryChatMessageHistory() return history_store[session_id] chatbot = RunnableWithMessageHistory( prompt | llm | StrOutputParser(), get_session_history, input_messages_key="input", history_messages_key="history", )

Initialize TruLens¶

Create a TruLens session and reset the default local database for a clean quickstart run.

In [ ]:

from trulens.core import TruSession
from trulens.core.database.connector.default import DefaultDBConnector

connector = DefaultDBConnector(
    database_url="sqlite:///conversation_evaluation.sqlite"
)
session = TruSession(connector=connector)
session.reset_database()

from trulens.core import TruSession from trulens.core.database.connector.default import DefaultDBConnector connector = DefaultDBConnector( database_url="sqlite:///conversation_evaluation.sqlite" ) session = TruSession(connector=connector) session.reset_database()

Define turn and conversation metrics¶

Answer Relevance selects one record's input and output. Coherence Across Turns uses .on_conversation() to select every ordered turn that shares a conversation_id.

In [ ]:

from trulens.core import Metric
from trulens.core import Selector
from trulens.providers.openai import OpenAI

provider = OpenAI(model_engine="gpt-4o-mini")

# One score for each independently recorded turn.
f_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

# One score for the ordered records in each conversation.
f_conversation_coherence = Metric(
    implementation=provider.coherence_across_turns,
    name="Coherence Across Turns",
).on_conversation()

from trulens.core import Metric from trulens.core import Selector from trulens.providers.openai import OpenAI provider = OpenAI(model_engine="gpt-4o-mini") # One score for each independently recorded turn. f_answer_relevance = Metric( implementation=provider.relevance_with_cot_reasons, name="Answer Relevance", selectors={ "prompt": Selector.select_record_input(), "response": Selector.select_record_output(), }, ) # One score for the ordered records in each conversation. f_conversation_coherence = Metric( implementation=provider.coherence_across_turns, name="Coherence Across Turns", ).on_conversation()

Wrap the chatbot with TruChain¶

In [ ]:

from trulens.apps.langchain import TruChain

tru_chatbot = TruChain(
    chatbot,
    app_name="Conversation Evaluation Quickstart",
    app_version="v1",
    feedbacks=[f_answer_relevance, f_conversation_coherence],
)

from trulens.apps.langchain import TruChain tru_chatbot = TruChain( chatbot, app_name="Conversation Evaluation Quickstart", app_version="v1", feedbacks=[f_answer_relevance, f_conversation_coherence], )

Record the first conversation¶

LangChain's session_id selects message history, while TruLens' conversation_id groups independently recorded turns for evaluation. Reusing the same value keeps those two scopes aligned, but they are separate mechanisms.

In [ ]:

# ── Conversation A: climate change ──────────────────────────────────────────
CONV_A = "conv-climate-001"

turns_a = [
    "What is the greenhouse effect?",
    "How does it relate to global warming?",
    "What are the most effective ways to reduce carbon emissions?",
]

with tru_chatbot(conversation_id=CONV_A) as recording_a:
    for turn in turns_a:
        chatbot.invoke(
            {"input": turn},
            config={"configurable": {"session_id": CONV_A}},
        )

# ── Conversation A: climate change ────────────────────────────────────────── CONV_A = "conv-climate-001" turns_a = [ "What is the greenhouse effect?", "How does it relate to global warming?", "What are the most effective ways to reduce carbon emissions?", ] with tru_chatbot(conversation_id=CONV_A) as recording_a: for turn in turns_a: chatbot.invoke( {"input": turn}, config={"configurable": {"session_id": CONV_A}}, )

Record the second conversation¶

Use a different history and conversation ID so the Python discussion is recorded and evaluated independently from the climate discussion.

In [ ]:

# ── Conversation B: Python programming ──────────────────────────────────────
CONV_B = "conv-python-002"

turns_b = [
    "What makes Python a good language for beginners?",
    "Can you explain list comprehensions with a simple example?",
    "When should I use a generator instead of a list?",
]

with tru_chatbot(conversation_id=CONV_B) as recording_b:
    for turn in turns_b:
        chatbot.invoke(
            {"input": turn},
            config={"configurable": {"session_id": CONV_B}},
        )

# ── Conversation B: Python programming ────────────────────────────────────── CONV_B = "conv-python-002" turns_b = [ "What makes Python a good language for beginners?", "Can you explain list comprehensions with a simple example?", "When should I use a generator instead of a list?", ] with tru_chatbot(conversation_id=CONV_B) as recording_b: for turn in turns_b: chatbot.invoke( {"input": turn}, config={"configurable": {"session_id": CONV_B}}, )

Wait for feedback¶

Conversation metrics are queued automatically when each recording context exits. Wait for both exact batches so the next cells can display persisted results.

In [ ]:

recording_a.retrieve_feedback_results()
recording_b.retrieve_feedback_results()

recording_a.retrieve_feedback_results() recording_b.retrieve_feedback_results()

View results¶

Overall leaderboard¶

In [ ]:

session.get_leaderboard()

session.get_leaderboard()

Inspect turn-level and conversation-level scores¶

Answer Relevance is populated for every turn. Coherence Across Turns is evaluated once for the ordered conversation, so its score is stored on the latest record in each conversation.

In [ ]:

records, feedback_columns = session.get_records_and_feedback(
    app_ids=[tru_chatbot.app_id]
)

records[
    [
        "conversation_id",
        "input",
        "output",
        "Answer Relevance",
        "Coherence Across Turns",
    ]
].sort_values(["conversation_id", "ts"])

records, feedback_columns = session.get_records_and_feedback( app_ids=[tru_chatbot.app_id] ) records[ [ "conversation_id", "input", "output", "Answer Relevance", "Coherence Across Turns", ] ].sort_values(["conversation_id", "ts"])

Summarize a per-turn metric by conversation¶

Answer Relevance is evaluated independently for each turn. Average it by conversation_id when you want one reporting value per conversation. Coherence Across Turns is already evaluated once on the ordered conversation and does not need this aggregation.

In [ ]:

answer_relevance_by_conversation = (
    records.dropna(subset=["conversation_id", "Answer Relevance"])
    .groupby("conversation_id", as_index=False)["Answer Relevance"]
    .mean()
    .rename(columns={"Answer Relevance": "Average Answer Relevance"})
)
answer_relevance_by_conversation

answer_relevance_by_conversation = ( records.dropna(subset=["conversation_id", "Answer Relevance"]) .groupby("conversation_id", as_index=False)["Answer Relevance"] .mean() .rename(columns={"Answer Relevance": "Average Answer Relevance"}) ) answer_relevance_by_conversation

Launch the TruLens dashboard¶

The dashboard's Conversation view lets you explore all turns of a conversation thread together, inspect per-turn scores, and compare conversations side by side.

In [ ]:

from trulens.dashboard import run_dashboard

run_dashboard(session)

from trulens.dashboard import run_dashboard run_dashboard(session)