trulens.feedback

trulens.feedback

Classes

GroundTruthAggregator

Bases: WithClassInfo, SerialModel

Attributes

tru_class_info instance-attribute

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

model_config class-attribute

model_config: ConfigDict = ConfigDict(
    arbitrary_types_allowed=True, extra="allow"
)

Aggregate benchmarking metrics for ground-truth-based evaluation on feedback functions.

Functions

__repr__

__repr__() -> str

Safe repr that handles circular references.

Pydantic's default __repr__ does not guard against circular references among model instances, which leads to RecursionError (see GitHub issue #1862). This override uses the same formatted_objects context-variable that __rich_repr__ uses so that already-visited objects are replaced with a short placeholder instead of recursing infinitely.

__rich_repr__

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load staticmethod

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate classmethod

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

register_custom_agg_func

register_custom_agg_func(
    name: str,
    func: Callable[
        [list[float], GroundTruthAggregator], float
    ],
) -> None

Register a custom aggregation function.

auc

auc(scores: list[float]) -> float

Calculate the area under the ROC curve. Can be used for meta-evaluation.

PARAMETER	DESCRIPTION
scores	scores returned by feedback function TYPE: List[float]

RETURNS	DESCRIPTION
float	Area under the ROC curve TYPE: float

kendall_tau

kendall_tau(scores: list[float] | list[list]) -> float

Calculate Kendall's tau. Can be used for meta-evaluation. Kendall’s tau is a measure of the correspondence between two rankings. Values close to 1 indicate strong agreement, values close to -1 indicate strong disagreement. This is the tau-b version of Kendall’s tau which accounts for ties.

PARAMETER	DESCRIPTION
scores	scores returned by feedback function TYPE: List[float]

RETURNS	DESCRIPTION
float	Kendall's tau TYPE: float

spearman_correlation

spearman_correlation(
    scores: list[float] | list[list],
) -> float

Calculate the Spearman correlation. Can be used for meta-evaluation. The Spearman correlation coefficient is a nonparametric measure of rank correlation (statistical dependence between the rankings of two variables).

PARAMETER	DESCRIPTION
scores	scores returned by feedback function TYPE: List[float]

RETURNS	DESCRIPTION
float	Spearman correlation TYPE: float

pearson_correlation

pearson_correlation(
    scores: list[float] | list[list],
) -> float

Calculate the Pearson correlation. Can be used for meta-evaluation. The Pearson correlation coefficient is a measure of the linear relationship between two variables.

PARAMETER	DESCRIPTION
scores	scores returned by feedback function TYPE: List[float]

RETURNS	DESCRIPTION
float	Pearson correlation TYPE: float

matthews_correlation

matthews_correlation(
    scores: list[float] | list[list],
) -> float

Calculate the Matthews correlation coefficient. Can be used for meta-evaluation. The Matthews correlation coefficient is used in machine learning as a measure of the quality of binary and multiclass classifications.

PARAMETER	DESCRIPTION
scores	scores returned by feedback function TYPE: List[float]

RETURNS	DESCRIPTION
float	Matthews correlation coefficient TYPE: float

cohens_kappa

cohens_kappa(
    scores: list[float] | list[list], threshold=0.5
) -> float

Computes Cohen's Kappa score between true labels and predicted scores.

Parameters: - true_labels (list): A list of true labels. - scores (list): A list of predicted labels or scores.

Returns: - float: Cohen's Kappa score.

recall

recall(scores: list[float] | list[list], threshold=0.5)

Calculates recall given true labels and model-generated scores.

Parameters: - scores (list of float): A list of model-generated scores (0 to 1.0). - threshold (float): The threshold to convert scores to binary predictions. Default is 0.5.

Returns: - float: The recall score.

precision

precision(scores: list[float] | list[list], threshold=0.5)

Calculates precision given true labels and model-generated scores.

Parameters: - scores (list of float): A list of model-generated scores (0 to 1.0). - threshold (float): The threshold to convert scores to binary predictions. Default is 0.5.

Returns: - float: The precision score.

f1_score

f1_score(scores: list[float] | list[list], threshold=0.5)

Calculates the F1 score given true labels and model-generated scores.

Parameters: - scores (list of float): A list of model-generated scores (0 to 1.0). - threshold (float): The threshold to convert scores to binary predictions. Default is 0.5.

Returns: - float: The F1 score.

brier_score

brier_score(scores: list[float] | list[list]) -> float

assess both calibration and sharpness of the probability estimates Args: scores (List[float]): relevance scores returned by feedback function Returns: float: Brier score

ece

ece(score_confidence_pairs: list[tuple[float]]) -> float

Calculate the expected calibration error. Can be used for meta-evaluation.

PARAMETER	DESCRIPTION
score_confidence_pairs	list of tuples of relevance scores and confidences returned by feedback function TYPE: List[Tuple[float]]

RETURNS	DESCRIPTION
float	Expected calibration error TYPE: float

mae

mae(scores: list[float] | list[list]) -> float

Calculate the mean absolute error. Can be used for meta-evaluation.

PARAMETER	DESCRIPTION
scores	scores returned by feedback function TYPE: List[float]

RETURNS	DESCRIPTION
float	Mean absolute error TYPE: float

GroundTruthAgreement

Bases: WithClassInfo, SerialModel

Measures Agreement against a Ground Truth.

Attributes

tru_class_info instance-attribute

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

conversation_id class-attribute instance-attribute

conversation_id: str | None = None

Optional conversation ID to scope ground truth lookups for memory recall.

Functions

__repr__

__repr__() -> str

Safe repr that handles circular references.

Pydantic's default __repr__ does not guard against circular references among model instances, which leads to RecursionError (see GitHub issue #1862). This override uses the same formatted_objects context-variable that __rich_repr__ uses so that already-visited objects are replaced with a short placeholder instead of recursing infinitely.

__rich_repr__

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load staticmethod

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate classmethod

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

__init__

__init__(
    ground_truth: (
        list[dict] | Callable | DataFrame | FunctionOrMethod
    ),
    provider: LLMProvider | None = None,
    bert_scorer: Optional[BERTScorer] = None,
    conversation_id: str | None = None,
    **kwargs
)

Measures Agreement against a Ground Truth.

Usage 1

from trulens.feedback import GroundTruthAgreement
from trulens.providers.openai import OpenAI
golden_set = [
    {"query": "who invented the lightbulb?", "expected_response": "Thomas Edison"},
    {"query": "¿quien invento la bombilla?", "expected_response": "Thomas Edison"}
]
ground_truth_collection = GroundTruthAgreement(golden_set, provider=OpenAI())

Usage 2

from trulens.feedback import GroundTruthAgreement
from trulens.providers.openai import OpenAI
from trulens.core.session import TruSession

session = TruSession()
ground_truth_dataset = session.get_ground_truths_by_dataset("hotpotqa") # assuming a dataset "hotpotqa" has been created and persisted in the DB

ground_truth_collection = GroundTruthAgreement(ground_truth_dataset, provider=OpenAI())

Usage 3

from snowflake.snowpark import Session
from trulens.feedback import GroundTruthAgreement
from trulens.providers.cortex import Cortex
ground_truth_imp = llm_app
response = llm_app(prompt)

snowflake_connection_parameters = {
    "account": os.environ["SNOWFLAKE_ACCOUNT"],
    "user": os.environ["SNOWFLAKE_USER"],
    "password": os.environ["SNOWFLAKE_USER_PASSWORD"],
    "database": os.environ["SNOWFLAKE_DATABASE"],
    "schema": os.environ["SNOWFLAKE_SCHEMA"],
    "warehouse": os.environ["SNOWFLAKE_WAREHOUSE"],
}

snowpark_session = Session.builder.configs(snowflake_connection_parameters).create()

ground_truth_collection = GroundTruthAgreement(
    ground_truth_imp,
    provider=Cortex(
        snowpark_session=snowpark_session,
        model_engine="mistral-7b",
    ),
)

PARAMETER	DESCRIPTION
ground_truth	A list of query/response pairs or a function, or a dataframe containing ground truth dataset, or callable that returns a ground truth string given a prompt string. TYPE: list[dict] | Callable | DataFrame | FunctionOrMethod
provider	The provider to use for agreement measures. TYPE: LLMProvider | None DEFAULT: None
bert_scorer	Internal Usage for DB serialization. TYPE: Optional[BERTScorer] DEFAULT: None
conversation_id	Optional conversation ID to scope ground truth lookups. When set, only ground truth entries matching this conversation_id will be considered for memory recall methods. TYPE: str | None DEFAULT: None

agreement_measure

agreement_measure(
    prompt: str, response: str
) -> float | tuple[float, dict[str, str]]

Uses OpenAI's Chat GPT Model. A function that measures similarity to ground truth. A second template is given to Chat GPT with a prompt that the original response is correct, and measures whether previous Chat GPT's response is similar.

Example

from trulens.core import Metric, Selector
from trulens.feedback import GroundTruthAgreement
from trulens.providers.openai import OpenAI

golden_set = [
    {"query": "who invented the lightbulb?", "expected_response": "Thomas Edison"},
    {"query": "¿quien invento la bombilla?", "expected_response": "Thomas Edison"}
]
ground_truth_collection = GroundTruthAgreement(golden_set, provider=OpenAI())

feedback = Metric(
    implementation=ground_truth_collection.agreement_measure,
    name="Agreement Measure",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
prompt	A text prompt to an agent. TYPE: str
response	The agent's response to the prompt. TYPE: str

RETURNS	DESCRIPTION
float	A value between 0 and 1. 0 being "not in agreement" and 1 being "in agreement". TYPE: float | tuple[float, dict[str, str]]
dict	with key 'ground_truth_response' TYPE: float | tuple[float, dict[str, str]]

ndcg_at_k

ndcg_at_k(
    query: str,
    retrieved_context_chunks: list[str],
    relevance_scores: list[float] | None = None,
    k: int | None = None,
) -> float

Compute NDCG@k for a given query and retrieved context chunks.

PARAMETER	DESCRIPTION
query	The input query string. TYPE: str
retrieved_context_chunks	List of retrieved context chunks. TYPE: List[str]
relevance_scores	Relevance scores for each retrieved chunk. TYPE: Optional[List[float]] DEFAULT: None
k	Rank position up to which to compute NDCG. If None, compute for all retrieved chunks. TYPE: Optional[int] DEFAULT: None

RETURNS	DESCRIPTION
float	Computed NDCG@k score. TYPE: float

precision_at_k

precision_at_k(
    query: str,
    retrieved_context_chunks: list[str],
    relevance_scores: list[float] | None = None,
    k: int | None = None,
) -> float

Compute Precision@k for a given query and retrieved context chunks, considering tie handling.

PARAMETER	DESCRIPTION
query	The input query string. TYPE: str
retrieved_context_chunks	List of retrieved context chunks. TYPE: List[str]
relevance_scores	Relevance scores for each retrieved chunk. TYPE: Optional[List[float]] DEFAULT: None
k	Rank position up to which to compute Precision. If None, compute for all retrieved chunks. TYPE: Optional[int] DEFAULT: None

RETURNS	DESCRIPTION
float	Computed Precision@k score. TYPE: float

recall_at_k

recall_at_k(
    query: str,
    retrieved_context_chunks: list[str],
    relevance_scores: list[float] | None = None,
    k: int | None = None,
) -> float

Compute Recall@k for a given query and retrieved context chunks, considering tie handling.

PARAMETER	DESCRIPTION
query	The input query string. TYPE: str
retrieved_context_chunks	List of retrieved context chunks. TYPE: List[str]
relevance_scores	Relevance scores for each retrieved chunk. TYPE: Optional[List[float]] DEFAULT: None
k	Rank position up to which to compute Recall. If None, compute for all retrieved chunks. TYPE: Optional[int] DEFAULT: None

RETURNS	DESCRIPTION
float	Computed Recall@k score. TYPE: float

mrr

mrr(
    query: str,
    retrieved_context_chunks: list[str],
    relevance_scores: list[float] | None = None,
) -> float

Compute Mean Reciprocal Rank (MRR) for a given query and retrieved context chunks.

PARAMETER	DESCRIPTION
query	The input query string. TYPE: str
retrieved_context_chunks	List of retrieved context chunks. TYPE: List[str]

RETURNS	DESCRIPTION
float	Computed MRR score. TYPE: float

ir_hit_rate

ir_hit_rate(
    query: str,
    retrieved_context_chunks: list[str],
    k: int | None = None,
) -> float

Compute IR Hit Rate (Hit Rate@k) for a given query and retrieved context chunks.

PARAMETER	DESCRIPTION
query	The input query string. TYPE: str
retrieved_context_chunks	List of retrieved context chunks. TYPE: List[str]
k	Rank position up to which to compute Hit Rate. If None, compute for all retrieved chunks. TYPE: Optional[int] DEFAULT: None

RETURNS	DESCRIPTION
float	Computed Hit Rate@k score. TYPE: float

absolute_error

absolute_error(
    prompt: str, response: str, score: float
) -> tuple[float, dict[str, float]]

Method to look up the numeric expected score from a golden set and take the difference.

Primarily used for evaluation of model generated feedback against human feedback

Example

from trulens.core import Metric, Selector
from trulens.feedback import GroundTruthAgreement
from trulens.providers.bedrock import Bedrock

golden_set = [
{"query": "How many stomachs does a cow have?", "expected_response": "Cows' diet relies primarily on grazing.", "expected_score": 0.4},
{"query": "Name some top dental floss brands", "expected_response": "I don't know", "expected_score": 0.8}
]

bedrock = Bedrock(
    model_id="amazon.nova-lite-v1:0", region_name="us-east-1"
)
ground_truth_collection = GroundTruthAgreement(golden_set, provider=bedrock)

f_groundtruth = Metric(
    implementation=ground_truth_collection.absolute_error,
    name="Absolute Error",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

bert_score

bert_score(
    prompt: str, response: str
) -> float | tuple[float, dict[str, str]]

Uses BERT Score. A function that that measures similarity to ground truth using bert embeddings.

Example

from trulens.core import Metric, Selector
from trulens.feedback import GroundTruthAgreement
from trulens.providers.openai import OpenAI
golden_set = [
    {"query": "who invented the lightbulb?", "expected_response": "Thomas Edison"},
    {"query": "¿quien invento la bombilla?", "expected_response": "Thomas Edison"}
]
ground_truth_collection = GroundTruthAgreement(golden_set, provider=OpenAI())

feedback = Metric(
    implementation=ground_truth_collection.bert_score,
    name="BERT Score",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
prompt	A text prompt to an agent. TYPE: str
response	The agent's response to the prompt. TYPE: str

RETURNS	DESCRIPTION
float	A value between 0 and 1. 0 being "not in agreement" and 1 being "in agreement". TYPE: float | tuple[float, dict[str, str]]
dict	with key 'ground_truth_response' TYPE: float | tuple[float, dict[str, str]]

bleu

bleu(
    prompt: str, response: str
) -> float | tuple[float, dict[str, str]]

Uses BLEU Score. A function that that measures similarity to ground truth using token overlap.

Example

from trulens.core import Metric, Selector
from trulens.feedback import GroundTruthAgreement
from trulens.providers.openai import OpenAI
golden_set = [
    {"query": "who invented the lightbulb?", "expected_response": "Thomas Edison"},
    {"query": "¿quien invento la bombilla?", "expected_response": "Thomas Edison"}
]
ground_truth_collection = GroundTruthAgreement(golden_set, provider=OpenAI())

feedback = Metric(
    implementation=ground_truth_collection.bleu,
    name="BLEU",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
prompt	A text prompt to an agent. TYPE: str
response	The agent's response to the prompt. TYPE: str

RETURNS	DESCRIPTION
float	A value between 0 and 1. 0 being "not in agreement" and 1 being "in agreement". TYPE: float | tuple[float, dict[str, str]]
dict	with key 'ground_truth_response' TYPE: float | tuple[float, dict[str, str]]

rouge

rouge(
    prompt: str, response: str
) -> float | tuple[float, dict[str, str]]

Uses BLEU Score. A function that that measures similarity to ground truth using token overlap.

PARAMETER	DESCRIPTION
prompt	A text prompt to an agent. TYPE: str
response	The agent's response to the prompt. TYPE: str

RETURNS	DESCRIPTION
float | tuple[float, dict[str, str]]	float: A value between 0 and 1. 0 being "not in agreement" and 1 being "in agreement".
float | tuple[float, dict[str, str]]	dict: with key 'ground_truth_response'

memory_recall

memory_recall(
    query: str,
    retrieved_memories: list[str],
    similarity_threshold: float = 1.0,
) -> float

Compute recall of expected memories against retrieved memories.

Evaluates how well a memory system retrieves relevant memories by comparing retrieved memory texts against ground truth expected memories. Unlike context retrieval metrics (precision_at_k, recall_at_k) which evaluate RAG chunk retrieval, this metric evaluates agent memory store recall — whether the right stored memories surface when needed.

Example

from trulens.core import Metric, Selector
from trulens.feedback import GroundTruthAgreement
from trulens.providers.openai import OpenAI

golden_set = [
    {
        "query": "What are the user's preferences?",
        "expected_memories": [
            "User prefers dark mode",
            "User likes Python",
        ],
        "conversation_id": "conv_123",
    }
]

gta = GroundTruthAgreement(
    golden_set,
    provider=OpenAI(),
    conversation_id="conv_123",
)

feedback = Metric(
    implementation=gta.memory_recall,
    name="Memory Recall",
    selectors={
        "query": Selector.select_record_input(),
        "retrieved_memories": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
query	The query string used to retrieve memories. TYPE: str
retrieved_memories	List of memory texts returned by the memory store. TYPE: list[str]
similarity_threshold	Threshold for text matching (0.0-1.0). Default 1.0 = exact match. Values < 1.0 enable fuzzy matching using SequenceMatcher ratio. TYPE: float DEFAULT: 1.0

RETURNS	DESCRIPTION
float	Recall score between 0.0 and 1.0, or np.nan if no ground truth found for the given query. TYPE: float

memory_mrr

memory_mrr(
    query: str,
    retrieved_memories: list[str],
    similarity_threshold: float = 1.0,
) -> float

Compute Mean Reciprocal Rank for memory retrieval.

Returns the reciprocal rank of the first relevant memory found in the retrieved results. This is useful for evaluating ranking quality — whether relevant memories appear early in results.

PARAMETER	DESCRIPTION
query	The query string used to retrieve memories. TYPE: str
retrieved_memories	List of memory texts returned by the memory store. TYPE: list[str]
similarity_threshold	Threshold for text matching (0.0-1.0). Default 1.0 = exact match. TYPE: float DEFAULT: 1.0

RETURNS	DESCRIPTION
float	MRR score between 0.0 and 1.0, or np.nan if no ground truth found for the given query. TYPE: float

Jury

Ensemble multiple LLM judges into a single feedback callable.

Jury wraps N provider instances, calls the same named method on each in parallel, and aggregates their scores using a configurable strategy. Because Jury.__call__ exposes the same parameter names as the underlying provider method, it plugs directly into Metric(implementation=jury) — no changes to Metric, Selector, or the evaluation pipeline are needed.

__call__ always returns (score, {"reason": ...}), matching the _with_cot_reasons convention so per-juror breakdowns flow into FeedbackCall.meta["reason"] and are visible in OTEL spans and the dashboard without any UI changes.

PARAMETER	DESCRIPTION
jurors	Non-empty list of LLMProvider instances. TYPE: list[Any]
method	Name of the feedback method to call on each juror, e.g. "relevance" or "groundedness_measure_with_cot_reasons". TYPE: str
aggregation	How to combine individual juror scores. Accepts a strategy name ("mean", "median", "trimmed_mean", "majority_vote", "weighted_mean") or any Callable[[list[float]], float]. Defaults to "mean". TYPE: str | Callable[[list[float]], float] DEFAULT: 'mean'
weights	Per-juror weights for "weighted_mean". Must have the same length as jurors. When a juror fails its weight is redistributed proportionally among the successful ones. TYPE: list[float] | None DEFAULT: None
threshold	Binarisation threshold for "majority_vote". Scores >= threshold count as a positive vote. Defaults to 0.5. On an exact tie falls back to median. TYPE: float DEFAULT: 0.5
max_workers	Maximum parallel threads. Defaults to len(jurors). TYPE: int | None DEFAULT: None

Example::

from trulens.core import Metric
from trulens.feedback.jury import Jury
from trulens.providers.openai import OpenAI
from trulens.providers.litellm import LiteLLM

jury = Jury(
    jurors=[
        OpenAI(model_engine="gpt-4o-mini"),
        OpenAI(model_engine="gpt-4.1-mini"),
        LiteLLM(model_engine="anthropic/claude-3-haiku-20240307"),
    ],
    method="relevance",
    aggregation="median",
)
m = Metric(implementation=jury, name="Jury Relevance").on_input().on_output()

Functions

__call__

__call__(
    *args: Any, **kwargs: Any
) -> tuple[float, dict[str, Any]]

Evaluate the same arguments in parallel across all jurors.

Always returns (score, {"reason": ...}), matching the _with_cot_reasons convention. Per-juror scores and any CoT explanations are embedded in the reason string so they appear in OTEL spans and the dashboard automatically.

LLMProvider

Bases: Provider

An LLM-based provider.

This is an abstract class and needs to be initialized as one of these:

• OpenAI and subclass AzureOpenAI.

• Bedrock.

• LiteLLM. LiteLLM provides an interface to a wide range of models.

• LangChain.

Attributes

tru_class_info instance-attribute

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

endpoint class-attribute instance-attribute

endpoint: Optional[Endpoint] = None

Endpoint supporting this provider.

Remote API invocations are handled by the endpoint.

reports_costs property

reports_costs: bool

Whether this provider reports tracked costs.

Providers that instrument their API calls to report token counts and dollar costs should override this to return True. When cost_budget is set on a :class:SamplingConfig and this property is False, a warning is emitted so the user knows the budget cannot be enforced for metrics using this provider.

Functions

__repr__

__repr__() -> str

Safe repr that handles circular references.

Pydantic's default __repr__ does not guard against circular references among model instances, which leads to RecursionError (see GitHub issue #1862). This override uses the same formatted_objects context-variable that __rich_repr__ uses so that already-visited objects are replaced with a short placeholder instead of recursing infinitely.

__rich_repr__

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load staticmethod

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate classmethod

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

generate_score

generate_score(
    system_prompt: str,
    user_prompt: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 10,
    temperature: float = 0.0,
) -> float | tuple[float, dict]

Base method to generate a score normalized to 0 to 1, used for evaluation.

PARAMETER	DESCRIPTION
system_prompt	A pre-formatted system prompt. TYPE: str
user_prompt	An optional user prompt. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value. TYPE: int DEFAULT: 0
max_score_val	The maximum score value. TYPE: int DEFAULT: 10
temperature	The temperature for the LLM response. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float | tuple[float, dict]	float | tuple[float, dict]: The normalized score on a 0-1 scale.
float | tuple[float, dict]	When the LLM returns a structured JSON response containing a score
float | tuple[float, dict]	key, returns a (score, reason_dict) tuple where reason_dict
float | tuple[float, dict]	contains the raw parsed JSON under a "reason" key. When the
float | tuple[float, dict]	response is a plain string, returns a bare float. Callers should
float | tuple[float, dict]	handle both forms, e.g.:: result = provider.generate_score(system_prompt="...") score = result[0] if isinstance(result, tuple) else result

generate_score_and_reasons

generate_score_and_reasons(
    system_prompt: str,
    user_prompt: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 10,
    temperature: float = 0.0,
) -> Tuple[float, Dict]

Base method to generate a score and reason, used for evaluation.

PARAMETER	DESCRIPTION
system_prompt	A pre-formatted system prompt. TYPE: str
user_prompt	An optional user prompt. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value. TYPE: int DEFAULT: 0
max_score_val	The maximum score value. TYPE: int DEFAULT: 10
temperature	The temperature for the LLM response. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing the normalized score on a 0-1 scale and reason metadata dictionary.

context_relevance

context_relevance(
    question: str,
    context: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the relevance of the context to the question.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.context_relevance,
    name="Context Relevance",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(
            collect_list=False
        ),
    },
    agg=np.mean,
)

PARAMETER	DESCRIPTION
question	A question being asked. TYPE: str
context	Context related to the question. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not relevant) and 1.0 (relevant). TYPE: float

context_relevance_with_cot_reasons

context_relevance_with_cot_reasons(
    question: str,
    context: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the relevance of the context to the question. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(
            collect_list=False
        ),
    },
    agg=np.mean,
)

PARAMETER	DESCRIPTION
question	A question being asked. TYPE: str
context	Context related to the question. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0 and 1. 0 being "not relevant" and 1 being "relevant". TYPE: Tuple[float, Dict]

citation_accuracy

citation_accuracy(
    response: str,
    context: Union[str, List[str]],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check whether the citations in a response are supported by the retrieved context, on a graded 0-3 scale.

Choosing between this and citation_attribution:

• Use citation_attribution when your pipeline emits explicit [N] markers and you want a binary pass/fail on misattribution: a claim cited to a passage that does not support it.
• Use citation_accuracy when citations are inline, prose, or otherwise not [N]-numbered, or when you want a graded score rather than a hard fail so you can track citation quality across runs.

Note that unlike citation_attribution, this metric penalizes missing citations: a claim that the context supports but that the response leaves uncited lowers the score. citation_attribution deliberately ignores uncited claims. Prefer that one if under-citation is acceptable in your pipeline and only misattribution matters.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.citation_accuracy,
    name="Citation Accuracy",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "response": Selector.select_record_output(),
        "context": Selector.select_context(
            collect_list=True
        ),
    },
    agg=np.mean,
)

PARAMETER	DESCRIPTION
response	The response containing citations to evaluate. TYPE: str
context	The retrieved context the citations should map to. A list of passages (as returned by Selector.select_context(collect_list=True)) is joined with blank lines into a single block; no [N] numbering is added, since this metric does not resolve numeric markers. A string is used as-is. TYPE: Union[str, List[str]]
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (citations inaccurate) and 1.0 (citations accurate). TYPE: float

citation_accuracy_with_cot_reasons

citation_accuracy_with_cot_reasons(
    response: str,
    context: Union[str, List[str]],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check whether the citations in a response are supported by the retrieved context. Also uses chain of thought methodology and emits the reasons.

Same check as citation_accuracy; see that method for how this metric compares to citation_attribution (format-agnostic and graded here, [N]-marker-based and binary there) and for the note that this metric penalizes missing citations while citation_attribution does not.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.citation_accuracy_with_cot_reasons,
    name="Citation Accuracy",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "response": Selector.select_record_output(),
        "context": Selector.select_context(
            collect_list=True
        ),
    },
    agg=np.mean,
)

PARAMETER	DESCRIPTION
response	The response containing citations to evaluate. TYPE: str
context	The retrieved context the citations should map to. A list of passages (as returned by Selector.select_context(collect_list=True)) is joined with blank lines into a single block; no [N] numbering is added, since this metric does not resolve numeric markers. A string is used as-is. TYPE: Union[str, List[str]]
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A value between 0.0 (citations inaccurate) and 1.0 (citations accurate), and a dictionary with the reasons for the score.

relevance

relevance(
    prompt: str,
    response: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the relevance of the response to a prompt.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.relevance,
    name="Answer Relevance",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
prompt	A text prompt to an agent. TYPE: str
response	The agent's response to the prompt. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0 and 1. 0 being "not relevant" and 1 being "relevant". TYPE: float

relevance_with_cot_reasons

relevance_with_cot_reasons(
    prompt: str,
    response: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion Model. A function that completes a template to check the relevance of the response to a prompt. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
prompt	A text prompt to an agent. TYPE: str
response	The agent's response to the prompt. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0 and 1. 0 being "not relevant" and 1 being "relevant". TYPE: Tuple[float, Dict]

sentiment

sentiment(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the sentiment of some text.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.sentiment,
    name="Sentiment",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate sentiment of. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0 and 1. 0 being "negative sentiment" and 1 being "positive sentiment". TYPE: float

sentiment_with_cot_reasons

sentiment_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the sentiment of some text. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.sentiment_with_cot_reasons,
    name="Sentiment",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	Text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (negative sentiment) and 1.0 (positive sentiment). TYPE: Tuple[float, Dict]

model_agreement

model_agreement(prompt: str, response: str) -> float

Uses chat completion model. A function that gives a chat completion model the same prompt and gets a response, encouraging truthfulness. A second template is given to the model with a prompt that the original response is correct, and measures whether previous chat completion response is similar.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.model_agreement,
    name="Model Agreement",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
prompt	A text prompt to an agent. TYPE: str
response	The agent's response to the prompt. TYPE: str

RETURNS	DESCRIPTION
float	A value between 0.0 (not in agreement) and 1.0 (in agreement). TYPE: float

conciseness

conciseness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the conciseness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.conciseness,
    name="Conciseness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate the conciseness of. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not concise) and 1.0 (concise). TYPE: float

conciseness_with_cot_reasons

conciseness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the conciseness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.conciseness_with_cot_reasons,
    name="Conciseness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

Args: text (str): The text to evaluate the conciseness of. criteria (Optional[str]): If provided, overrides the default criteria for evaluation. Defaults to None. additional_instructions (Optional[str]): If provided, adds instructions to default criteria for the judge to follow. Defaults to None. min_score_val (int): The minimum score value used by the LLM before normalization. Defaults to 0. max_score_val (int): The maximum score value used by the LLM before normalization. Defaults to 3. temperature (float): The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0.

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not concise) and 1.0 (concise) and a dictionary containing the reasons for the evaluation.

correctness

correctness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the correctness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.correctness,
    name="Correctness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	A prompt to an agent. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not correct) and 1.0 (correct). TYPE: float

correctness_with_cot_reasons

correctness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the correctness of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.correctness_with_cot_reasons,
    name="Correctness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	Text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not correct) and 1.0 (correct) and a dictionary containing the reasons for the evaluation.

coherence

coherence(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the coherence of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.coherence,
    name="Coherence",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not coherent) and 1.0 (coherent). TYPE: float

coherence_with_cot_reasons

coherence_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the coherence of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.coherence_with_cot_reasons,
    name="Coherence",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not coherent) and 1.0 (coherent) and a dictionary containing the reasons for the evaluation.

harmfulness

harmfulness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the harmfulness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.harmfulness,
    name="Harmfulness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not harmful) and 1.0 (harmful)". TYPE: float

harmfulness_with_cot_reasons

harmfulness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the harmfulness of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.harmfulness_with_cot_reasons,
    name="Harmfulness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not harmful) and 1.0 (harmful) and a dictionary containing the reasons for the evaluation.

maliciousness

maliciousness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the maliciousness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.maliciousness,
    name="Maliciousness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not malicious) and 1.0 (malicious). TYPE: float

maliciousness_with_cot_reasons

maliciousness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the maliciousness of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.maliciousness_with_cot_reasons,
    name="Maliciousness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not malicious) and 1.0 (malicious) and a dictionary containing the reasons for the evaluation.

helpfulness

helpfulness(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the helpfulness of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.helpfulness,
    name="Helpfulness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not helpful) and 1.0 (helpful). TYPE: float

helpfulness_with_cot_reasons

helpfulness_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the helpfulness of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.helpfulness_with_cot_reasons,
    name="Helpfulness",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not helpful) and 1.0 (helpful) and a dictionary containing the reasons for the evaluation.

controversiality

controversiality(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the controversiality of some text. Prompt credit to Langchain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.controversiality,
    name="Controversiality",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not controversial) and 1.0 (controversial). TYPE: float

controversiality_with_cot_reasons

controversiality_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the controversiality of some text. Prompt credit to Langchain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.controversiality_with_cot_reasons,
    name="Controversiality",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not controversial) and 1.0 (controversial) and a dictionary containing the reasons for the evaluation.

misogyny

misogyny(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the misogyny of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.misogyny,
    name="Misogyny",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not misogynistic) and 1.0 (misogynistic). TYPE: float

misogyny_with_cot_reasons

misogyny_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the misogyny of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.misogyny_with_cot_reasons,
    name="Misogyny",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not misogynistic) and 1.0 (misogynistic) and a dictionary containing the reasons for the evaluation.

criminality

criminality(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the criminality of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.criminality,
    name="Criminality",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not criminal) and 1.0 (criminal). TYPE: float

criminality_with_cot_reasons

criminality_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the criminality of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.criminality_with_cot_reasons,
    name="Criminality",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not criminal) and 1.0 (criminal) and a dictionary containing the reasons for the evaluation.

insensitivity

insensitivity(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check the insensitivity of some text. Prompt credit to LangChain Eval.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.insensitivity,
    name="Insensitivity",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (not insensitive) and 1.0 (insensitive). TYPE: float

insensitivity_with_cot_reasons

insensitivity_with_cot_reasons(
    text: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check the insensitivity of some text. Prompt credit to LangChain Eval. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.insensitivity_with_cot_reasons,
    name="Insensitivity",
    criteria=criteria,
    additional_instructions=additional_instructions,
    selectors={
        "text": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
text	The text to evaluate. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not insensitive) and 1.0 (insensitive) and a dictionary containing the reasons for the evaluation.

comprehensiveness_with_cot_reasons

comprehensiveness_with_cot_reasons(
    source: str,
    summary: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that tries to distill main points and compares a summary against those main points. This feedback function only has a chain of thought implementation as it is extremely important in function assessment.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.comprehensiveness_with_cot_reasons,
    name="Comprehensiveness",
    selectors={
        "source": Selector.select_record_input(),
        "summary": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
source	Text corresponding to source material. TYPE: str
summary	Text corresponding to a summary. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (not comprehensive) and 1.0 (comprehensive) and a dictionary containing the reasons for the evaluation.

summarization_with_cot_reasons

summarization_with_cot_reasons(
    source: str, summary: str
) -> Tuple[float, Dict]

Summarization is deprecated in place of comprehensiveness. This function is no longer implemented.

stereotypes

stereotypes(
    prompt: str,
    response: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> float

Uses chat completion model. A function that completes a template to check adding assumed stereotypes in the response when not present in the prompt.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.stereotypes,
    name="Stereotypes",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
prompt	A text prompt to an agent. TYPE: str
response	The agent's response to the prompt. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between 0.0 (no stereotypes assumed) and 1.0 (stereotypes assumed). TYPE: float

stereotypes_with_cot_reasons

stereotypes_with_cot_reasons(
    prompt: str,
    response: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Uses chat completion model. A function that completes a template to check adding assumed stereotypes in the response when not present in the prompt. Also uses chain of thought methodology and emits the reasons.

Example

from trulens.core import Metric, Selector
feedback = Metric(
    implementation=provider.stereotypes_with_cot_reasons,
    name="Stereotypes",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
prompt	A text prompt to an agent. TYPE: str
response	The agent's response to the prompt. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. TYPE: Optional[str] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A tuple containing a value between 0.0 (no stereotypes assumed) and 1.0 (stereotypes assumed) and a dictionary containing the reasons for the evaluation.

citation_attribution

citation_attribution(
    question: str,
    source: Union[str, List[str]],
    statement: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 1,
    temperature: float = 0.0,
    **kwargs
) -> float

Check citation-attribution faithfulness of a cited answer.

Unlike groundedness (does the source support the statement somewhere), this checks attribution: whether each [N] citation marker in the statement points to the SOURCE passage that supports the specific claim it is attached to. It catches misattribution: a claim cited to passage [A] that does not support it, even though some other passage [B] in the source would.

Example

from trulens.core import Metric, Selector

f_citation = Metric(
    implementation=provider.citation_attribution,
    name="Citation Attribution",
    criteria=criteria,
    additional_instructions=additional_instructions,
    examples=examples,
    selectors={
        "question": Selector.select_record_input(),
        "source": Selector.select_context(collect_list=True),
        "statement": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
question	The question being answered. TYPE: str
source	The retrieved passages. A list is numbered [1] ..., [2] ... so the statement's [N] markers resolve; a pre-numbered string is used as-is. TYPE: Union[str, List[str]]
statement	The answer, containing [N] citation markers. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value. Defaults to 1. TYPE: int DEFAULT: 1
temperature	The temperature for the LLM response. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
float	A value between min_score_val and max_score_val, normalized to 0.0 (a claim is misattributed) to 1.0 (every claim's citation points to a passage that supports it). TYPE: float

citation_attribution_with_cot_reasons

citation_attribution_with_cot_reasons(
    question: str,
    source: Union[str, List[str]],
    statement: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    min_score_val: int = 0,
    max_score_val: int = 1,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, Dict]

Citation-attribution faithfulness with chain-of-thought reasons.

Same check as citation_attribution, but also returns the reasoning for the verdict (which [N] marker, if any, is misattributed).

PARAMETER	DESCRIPTION
question	The question being answered. TYPE: str
source	The retrieved passages (a list is numbered for [N] resolution). TYPE: Union[str, List[str]]
statement	The answer, containing [N] citation markers. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
min_score_val	The minimum score value. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value. Defaults to 1. TYPE: int DEFAULT: 1
temperature	The temperature for the LLM response. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, Dict]	Tuple[float, Dict]: A score between 0.0 and 1.0 and a dictionary with the reasons for the evaluation.

groundedness_measure_with_cot_reasons

groundedness_measure_with_cot_reasons(
    source: str,
    statement: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[str] = None,
    groundedness_configs: Optional[
        GroundednessConfigs
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, dict]

A measure to track if the source material supports each sentence in the statement using an LLM provider.

The statement will first be split by a tokenizer into its component sentences.

Then, trivial statements are eliminated so as to not dilute the evaluation. Note that if all statements are filtered out as trivial, returns 0.0 with a reason indicating no non-trivial statements were found.

The LLM will process each statement, using chain of thought methodology to emit the reasons.

Abstentions will be considered as grounded.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(
            collect_list=True
        ),
        "statement": Selector.select_record_output(),
    },
)

To further explain how the function works under the hood, consider the statement:

"Hi. I'm here to help. The university of Washington is a public research university. UW's connections to major corporations in Seattle contribute to its reputation as a hub for innovation and technology"

The function will split the statement into its component sentences:

1. "Hi."
2. "I'm here to help."
3. "The university of Washington is a public research university."
4. "UW's connections to major corporations in Seattle contribute to its reputation as a hub for innovation and technology"

Next, trivial statements are removed, leaving only:

1. "The university of Washington is a public research university."
2. "UW's connections to major corporations in Seattle contribute to its reputation as a hub for innovation and technology"

The LLM will then process the statement, to assess the groundedness of the statement.

For the sake of this example, the LLM will grade the groundedness of one statement as 10, and the other as 0.

Then, the scores are normalized, and averaged to give a final groundedness score of 0.5.

PARAMETER	DESCRIPTION
source	The source that should support the statement. TYPE: str
statement	The statement to check groundedness. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. TYPE: Optional[str] DEFAULT: None
examples	Optional examples to guide the evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
groundedness_configs	Configuration for groundedness evaluation. Defaults to None. TYPE: Optional[GroundednessConfigs] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, dict]	Tuple[float, dict]: A tuple containing a value between 0.0 (not grounded) and 1.0 (grounded) and a dictionary containing the reasons for the evaluation.

qs_relevance

qs_relevance(*args, **kwargs)

Deprecated. Use relevance instead.

qs_relevance_with_cot_reasons

qs_relevance_with_cot_reasons(*args, **kwargs)

Deprecated. Use relevance_with_cot_reasons instead.

groundedness_measure_with_cot_reasons_consider_answerability

groundedness_measure_with_cot_reasons_consider_answerability(
    source: str,
    statement: str,
    question: str,
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[List[str]] = None,
    groundedness_configs: Optional[
        GroundednessConfigs
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    **kwargs
) -> Tuple[float, dict]

A measure to track if the source material supports each sentence in the statement using an LLM provider.

The statement will first be split by a tokenizer into its component sentences.

Then, trivial statements are eliminated so as to not dilute the evaluation. Note that if all statements are filtered out as trivial, returns 0.0 with a reason indicating no non-trivial statements were found.

The LLM will process each statement, using chain of thought methodology to emit the reasons.

In the case of abstentions, such as 'I do not know', the LLM will be asked to consider the answerability of the question given the source material.

If the question is considered answerable, abstentions will be considered as not grounded and punished with low scores. Otherwise, unanswerable abstentions will be considered grounded.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons_consider_answerability,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(
            collect_list=True
        ),
        "statement": Selector.select_record_output(),
        "question": Selector.select_record_input(),
    },
)

PARAMETER	DESCRIPTION
source	The source that should support the statement. TYPE: str
statement	The statement to check groundedness. TYPE: str
question	The question to check answerability. TYPE: str
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. TYPE: Optional[str] DEFAULT: None
examples	Optional examples to guide the evaluation. Defaults to None. TYPE: Optional[List[str]] DEFAULT: None
groundedness_configs	Configuration for groundedness evaluation. Defaults to None. TYPE: Optional[GroundednessConfigs] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
Tuple[float, dict]	Tuple[float, dict]: A tuple containing a value between 0.0 (not grounded) and 1.0 (grounded) and a dictionary containing the reasons for the evaluation.

logical_consistency_with_cot_reasons

logical_consistency_with_cot_reasons(
    trace: Union[Trace, str],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[
        List[Tuple[Dict[str, str], int]]
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    enable_trace_compression: bool = True,
    **kwargs
) -> Tuple[float, Dict]

Evaluate the quality of an agentic trace using a rubric focused on logical consistency and reasoning.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_logical_consistency = Metric(
    implementation=provider.logical_consistency_with_cot_reasons,
    name="Logical Consistency",
    selectors={
        "trace": Selector(trace_level=True),
    },
)

PARAMETER	DESCRIPTION
trace	The trace to evaluate (e.g., as a JSON string or formatted log). TYPE: Union[Trace, str]
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples for evaluation. Defaults to None. TYPE: Optional[List[Tuple[Dict[str, str], int]]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0
enable_trace_compression	Whether to compress the trace data to reduce token usage. When True (default), traces are compressed to preserve essential information while removing redundant data. Set to False to use full, uncompressed traces. This parameter is only available for feedback functions that take 'trace' as input. Defaults to True. TYPE: bool DEFAULT: True

Returns: Tuple[float, Dict]: A tuple containing a value between 0.0 (no logical consistency) and 1.0 (complete logical consistency) and a dictionary containing the reasons for the evaluation.

execution_efficiency_with_cot_reasons

execution_efficiency_with_cot_reasons(
    trace: Union[Trace, str],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[
        List[Tuple[Dict[str, str], int]]
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    enable_trace_compression: bool = True,
    **kwargs
) -> Tuple[float, Dict]

Evaluate the quality of an agentic execution using a rubric focused on execution efficiency.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_execution_efficiency = Metric(
    implementation=provider.execution_efficiency_with_cot_reasons,
    name="Execution Efficiency",
    selectors={
        "trace": Selector(trace_level=True),
    },
)

PARAMETER	DESCRIPTION
trace	The trace to evaluate (e.g., as a JSON string or formatted log). TYPE: Union[Trace, str]
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples for evaluation. Defaults to None. TYPE: Optional[List[Tuple[Dict[str, str], int]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0
enable_trace_compression	Whether to compress the trace data to reduce token usage. When True (default), traces are compressed to preserve essential information while removing redundant data. Set to False to use full, uncompressed traces. This parameter is only available for feedback functions that take 'trace' as input. Defaults to True. TYPE: bool DEFAULT: True

Returns: Tuple[float, Dict]: A tuple containing a value between 0.0 (highly inefficient workflow) and 1.0 (highly streamlined/optimized workflow) and a dictionary containing the reasons for the evaluation.

plan_adherence_with_cot_reasons

plan_adherence_with_cot_reasons(
    trace: Union[Trace, str],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[
        List[Tuple[Dict[str, str], int]]
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    enable_trace_compression: bool = True,
    **kwargs
) -> Tuple[float, Dict]

Evaluate the quality of an agentic trace using a rubric focused on execution adherence to the plan.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_plan_adherence = Metric(
    implementation=provider.plan_adherence_with_cot_reasons,
    name="Plan Adherence",
    selectors={
        "trace": Selector(trace_level=True),
    },
)

PARAMETER	DESCRIPTION
trace	The trace to evaluate (e.g., as a JSON string or formatted log). TYPE: Union[Trace, str]
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples for evaluation. Defaults to None. TYPE: Optional[List[Tuple[Dict[str, str], int]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0
enable_trace_compression	Whether to compress the trace data to reduce token usage. When True (default), traces are compressed to preserve essential information while removing redundant data. Set to False to use full, uncompressed traces. This parameter is only available for feedback functions that take 'trace' as input. Defaults to True. TYPE: bool DEFAULT: True

Returns: Tuple[float, Dict]: A tuple containing a value between 0.0 (execution did not follow plan) and 1.0 (execution followed plan exactly) and a dictionary containing the reasons for the evaluation.

plan_quality_with_cot_reasons

plan_quality_with_cot_reasons(
    trace: Union[Trace, str],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[
        List[Tuple[Dict[str, str], int]]
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    enable_trace_compression: bool = True,
    **kwargs
) -> Tuple[float, Dict]

Evaluate the quality of an agentic system's plan.

Example

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_plan_quality = Metric(
    implementation=provider.plan_quality_with_cot_reasons,
    name="Plan Quality",
    selectors={
        "trace": Selector(trace_level=True),
    },
)

PARAMETER	DESCRIPTION
trace	The trace to evaluate (e.g., as a JSON string or formatted log). TYPE: Union[Trace, str]
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples for evaluation. Defaults to None. TYPE: Optional[List[Tuple[Dict[str, str], int]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0
enable_trace_compression	Whether to compress the trace data to reduce token usage. When True (default), traces are compressed to preserve essential information while removing redundant data. Set to False to use full, uncompressed traces. This parameter is only available for feedback functions that take 'trace' as input. Defaults to True. TYPE: bool DEFAULT: True

Returns: Tuple[float, Dict]: A tuple containing a value between 0.0 (poor plan quality) and 1.0 (excellent plan quality) and a dictionary containing the reasons for the evaluation.

tool_selection_with_cot_reasons

tool_selection_with_cot_reasons(
    trace: Union[Trace, str],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[
        List[Tuple[Dict[str, str], int]]
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    enable_trace_compression: bool = True,
    **kwargs
) -> Tuple[float, Dict]

Evaluate the quality of an agentic trace using a rubric focused on tool selection. Example:

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_tool_selection = Metric(
    implementation=provider.tool_selection_with_cot_reasons,
    name="Tool Selection",
    selectors={
        "trace": Selector(trace_level=True),
    },
)

PARAMETER	DESCRIPTION
trace	The trace to evaluate (e.g., as a JSON string or formatted log). TYPE: Union[Trace, str]
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples for evaluation. Defaults to None. TYPE: Optional[List[Tuple[Dict[str, str], int]]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0
enable_trace_compression	Whether to compress the trace data to reduce token usage. When True (default), traces are compressed to preserve essential information while removing redundant data. Set to False to use full, uncompressed traces. This parameter is only available for feedback functions that take 'trace' as input. Defaults to True. TYPE: bool DEFAULT: True

Returns: Tuple[float, Dict]: A tuple containing a value between 0.0 (poor tool selection) and 1.0 (excellent tool selection) and a dictionary containing the reasons for the evaluation.

tool_calling_with_cot_reasons

tool_calling_with_cot_reasons(
    trace: Union[Trace, str],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[
        List[Tuple[Dict[str, str], int]]
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    enable_trace_compression: bool = True,
    **kwargs
) -> Tuple[float, Dict]

Evaluate the quality of an agentic trace using a rubric focused on tool calling. Example:

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_tool_calling = Metric(
    implementation=provider.tool_calling_with_cot_reasons,
    name="Tool Calling",
    selectors={
        "trace": Selector(trace_level=True),
    },
)

PARAMETER	DESCRIPTION
trace	The trace to evaluate (e.g., as a JSON string or formatted log). TYPE: Union[Trace, str]
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples for evaluation. Defaults to None. TYPE: Optional[List[Tuple[Dict[str, str], int]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0
enable_trace_compression	Whether to compress the trace data to reduce token usage. When True (default), traces are compressed to preserve essential information while removing redundant data. Set to False to use full, uncompressed traces. This parameter is only available for feedback functions that take 'trace' as input. Defaults to True. TYPE: bool DEFAULT: True

Returns: Tuple[float, Dict]: A tuple containing a value between 0.0 (poor tool calling) and 1.0 (excellent tool calling) and a dictionary containing the reasons for the evaluation.

tool_quality_with_cot_reasons

tool_quality_with_cot_reasons(
    trace: Union[Trace, str],
    criteria: Optional[str] = None,
    additional_instructions: Optional[str] = None,
    examples: Optional[
        List[Tuple[Dict[str, str], int]]
    ] = None,
    min_score_val: int = 0,
    max_score_val: int = 3,
    temperature: float = 0.0,
    enable_trace_compression: bool = True,
    **kwargs
) -> Tuple[float, Dict]

Evaluate the quality of an agentic trace using a rubric focused on tool quality. Example:

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

f_tool_quality = Metric(
    implementation=provider.tool_quality_with_cot_reasons,
    name="Tool Quality",
    selectors={
        "trace": Selector(trace_level=True),
    },
)

PARAMETER	DESCRIPTION
trace	The trace to evaluate (e.g., as a JSON string or formatted log). TYPE: Union[Trace, str]
criteria	If provided, overrides the default criteria for evaluation. Defaults to None. TYPE: Optional[str] DEFAULT: None
additional_instructions	If provided, adds instructions to default criteria for the judge to follow. Defaults to None. TYPE: Optional[str] DEFAULT: None
examples	Optional few-shot examples for evaluation. Defaults to None. TYPE: Optional[List[Tuple[Dict[str, str], int]] DEFAULT: None
min_score_val	The minimum score value used by the LLM before normalization. Defaults to 0. TYPE: int DEFAULT: 0
max_score_val	The maximum score value used by the LLM before normalization. Defaults to 3. TYPE: int DEFAULT: 3
temperature	The temperature for the LLM response, which might have impact on the confidence level of the evaluation. Defaults to 0.0. TYPE: float DEFAULT: 0.0
enable_trace_compression	Whether to compress the trace data to reduce token usage. When True (default), traces are compressed to preserve essential information while removing redundant data. Set to False to use full, uncompressed traces. This parameter is only available for feedback functions that take 'trace' as input. Defaults to True. TYPE: bool DEFAULT: True

Returns: Tuple[float, Dict]: A tuple containing a value between 0.0 (poor tool quality) and 1.0 (excellent tool quality) and a dictionary containing the reasons for the evaluation.

conversation_helpfulness

conversation_helpfulness(
    records: Union[List[Any], str], temperature: float = 0.0
) -> float

Evaluate helpfulness across a multi-turn conversation.

topic_adherence

topic_adherence(
    records: Union[List[Any], str],
    reference_topics: List[str],
    temperature: float = 0.0,
) -> float

Evaluate topic adherence across a multi-turn conversation.

agent_goal_accuracy

agent_goal_accuracy(
    records: Union[List[Any], str],
    reference_goal: Optional[str] = None,
    temperature: float = 0.0,
) -> float

Evaluate whether an agent fulfilled the conversation goal.

coherence_across_turns

coherence_across_turns(
    records: Union[List[Any], str], temperature: float = 0.0
) -> float

Evaluate logical coherence across conversation turns.

FewShotOptimizer

Select the best few-shot examples for a TruLens feedback function.

The optimizer works by:

1. Iterating over candidates one at a time in parallel rounds.
2. For each candidate, calling feedback_fn on every sample in eval_dataset with that candidate injected as a few-shot example.
3. Computing the target metric (e.g., Pearson correlation, F1, precision, recall, Cohen's kappa) between predicted and ground-truth scores.
4. Greedily selecting the n_examples candidates with the highest metric improvement (greedy forward selection).

Parameters

feedback_fn: A callable that accepts the keyword arguments defined in candidates plus an optional examples: str keyword argument. It must return a float in [0, 1]. Must be thread-safe; called concurrently when max_workers > 1. Typically a bound method on a :class:trulens.feedback.LLMProvider subclass, e.g. provider.relevance. candidates: Pool of demonstration examples to select from. Each entry is a (feedback_kwargs, ground_truth_score) pair where ground_truth_score is a float in [0, 1]. eval_dataset: Held-out labeled examples used to measure how well a candidate set helps the judge. Should be disjoint from candidates to avoid overfitting. n_examples: Maximum number of examples to include in the final prompt. Defaults to 3. format_sep: Separator inserted between formatted examples when building the examples string passed to feedback_fn. Defaults to "\n\n". metric: Evaluation metric to optimize. Supported metrics: "pearson", "spearman", "precision", "recall", "f1", "cohens_kappa", "accuracy", "mae". Defaults to "pearson". metric_threshold: Threshold used for binarizing scores when computing classification metrics (precision, recall, f1, cohens_kappa, accuracy). Defaults to 0.5. max_workers: Maximum number of thread workers used to evaluate candidate sets in parallel. Defaults to None (utilizes ThreadPoolExecutor's default).

Functions

optimize

optimize() -> OptimizeResult

Run greedy forward selection and return the best example subset.

Returns

OptimizeResult Contains the selected examples, overall evaluation score, and per-candidate scores.

Raises

RuntimeError If feedback_fn raises an exception for every candidate on the first eval sample (likely a misconfigured provider).

format_examples

format_examples(examples: list[LabeledExample]) -> str

Serialise a list of labeled examples into the string format expected by feedback_fn's examples parameter.

Each example is rendered as a bullet showing the input kwargs and the expected score, separated by :attr:format_sep.

Parameters

examples: Subset of labeled examples to format, typically the output of :meth:optimize.

Returns

str A human-readable string ready to be passed as feedback_fn(..., examples=<return_value>).

OptimizeResult dataclass

Outcome of a :meth:FewShotOptimizer.optimize run.

Attributes

best_examples: The subset of candidates selected by the optimizer, each paired with its ground-truth score. correlation: Evaluation score (e.g. Pearson correlation or selected metric score) achieved on eval_dataset when using best_examples. Higher is better. None if fewer than two eval samples were available. candidate_scores: Mapping from candidate index → metric score achieved when that candidate was included in the prompt. Useful for debugging. metric_name: Name of the metric used for optimization (e.g. "pearson", "f1"). metric_score: The metric score achieved by best_examples.

SchemaValidator

Bases: WithClassInfo, SerialModel

Non-LLM feedback functions for validating LLM output against a schema.

Accepts either a JSON schema dict (requires jsonschema) or a Pydantic model class. Each method returns 1.0 when the output is valid and 0.0 otherwise, along with a metadata dict that contains any validation errors.

Example — JSON schema dict:

from trulens.feedback.schema_validator import SchemaValidator
from trulens.core.metric.metric import Metric

schema = {
    "type": "object",
    "properties": {"answer": {"type": "string"}},
    "required": ["answer"],
}
validator = SchemaValidator(schema=schema)
f = Metric(validator.validate_json).on_output()

Example — Pydantic model:

import pydantic
from trulens.feedback.schema_validator import SchemaValidator
from trulens.core.metric.metric import Metric

class MyOutput(pydantic.BaseModel):
    answer: str
    score: float

validator = SchemaValidator(schema=MyOutput)
f = Metric(validator.validate_json).on_output()

Attributes

tru_class_info instance-attribute

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

Functions

__repr__

__repr__() -> str

Safe repr that handles circular references.

Pydantic's default __repr__ does not guard against circular references among model instances, which leads to RecursionError (see GitHub issue #1862). This override uses the same formatted_objects context-variable that __rich_repr__ uses so that already-visited objects are replaced with a short placeholder instead of recursing infinitely.

__rich_repr__

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load staticmethod

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate classmethod

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

__init__

__init__(schema: _SchemaType, **kwargs)

Create a SchemaValidator.

PARAMETER	DESCRIPTION
schema	Either a JSON schema dict or a Pydantic BaseModel class (not an instance). TYPE: _SchemaType

validate_json

validate_json(output: str) -> tuple[float, dict[str, str]]

Validate that output is valid JSON conforming to the schema.

Returns 1.0 when valid, 0.0 otherwise. The accompanying metadata dict always contains an "explanation" key describing the outcome (or the first validation error).

PARAMETER	DESCRIPTION
output	The string to validate. It must be parseable as JSON. TYPE: str

RETURNS	DESCRIPTION
float	A (score, metadata) tuple compatible with TruLens feedback
dict[str, str]	infrastructure.

validate_json_partial

validate_json_partial(
    output: str, required_keys: list | None = None
) -> tuple[float, dict[str, str]]

Validate that output is valid JSON and optionally check for keys.

This is a lighter-weight check: it verifies that output parses as a JSON object and, when required_keys is provided, that each key is present at the top level. The full schema is not consulted, making this useful for streaming or partial outputs.

PARAMETER	DESCRIPTION
output	The string to validate. TYPE: str
required_keys	Optional list of keys that must exist at the top level of the parsed object. TYPE: list | None DEFAULT: None

RETURNS	DESCRIPTION
tuple[float, dict[str, str]]	A (score, metadata) tuple.

Embeddings

Bases: WithClassInfo, SerialModel

Embedding related feedback function implementations.

Attributes

tru_class_info instance-attribute

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

Functions

__repr__

__repr__() -> str

Safe repr that handles circular references.

Pydantic's default __repr__ does not guard against circular references among model instances, which leads to RecursionError (see GitHub issue #1862). This override uses the same formatted_objects context-variable that __rich_repr__ uses so that already-visited objects are replaced with a short placeholder instead of recursing infinitely.

__rich_repr__

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load staticmethod

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate classmethod

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

__init__

__init__(embed_model: BaseEmbedding)

Instantiates embeddings for feedback functions.

Example

Below is just one example. Embedders from LlamaIndex are supported: https://docs.llamaindex.ai/en/latest/module_guides/models/embeddings/

from llama_index.embeddings.openai import OpenAIEmbedding
from trulens.feedback.embeddings import Embeddings

embed_model = OpenAIEmbedding()

f_embed = Embedding(embed_model=embed_model)

PARAMETER	DESCRIPTION
embed_model	Supports embedders from LlamaIndex: https://docs.llamaindex.ai/en/latest/module_guides/models/embeddings/ TYPE: BaseEmbedding

cosine_distance

cosine_distance(
    query: str, document: str
) -> Union[float, Tuple[float, Dict[str, str]]]

Runs cosine distance on the query and document embeddings

Example

Below is just one example. Embedders from LlamaIndex are supported: https://docs.llamaindex.ai/en/latest/module_guides/models/embeddings/

from llama_index.embeddings.openai import OpenAIEmbedding
from trulens.feedback.embeddings import Embeddings

embed_model = OpenAIEmbedding()

# Create the feedback function
from trulens.core import Metric, Selector
f_embed = feedback.Embeddings(embed_model=embed_model)
f_embed_dist = Metric(
    implementation=f_embed.cosine_distance,
    name="Cosine Distance",
    selectors={
        "query": Selector.select_record_input(),
        "document": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
query	A text prompt to a vector DB. TYPE: str
document	The document returned from the vector DB. TYPE: str

RETURNS	DESCRIPTION
float	the embedding vector distance TYPE: Union[float, Tuple[float, Dict[str, str]]]

manhattan_distance

manhattan_distance(
    query: str, document: str
) -> Union[float, Tuple[float, Dict[str, str]]]

Runs L1 distance on the query and document embeddings

Example

Below is just one example. Embedders from LlamaIndex are supported: https://docs.llamaindex.ai/en/latest/module_guides/models/embeddings/

from llama_index.embeddings.openai import OpenAIEmbedding
from trulens.feedback.embeddings import Embeddings

embed_model = OpenAIEmbedding()

# Create the feedback function
from trulens.core import Metric, Selector
f_embed = feedback.Embeddings(embed_model=embed_model)
f_embed_dist = Metric(
    implementation=f_embed.manhattan_distance,
    name="Manhattan Distance",
    selectors={
        "query": Selector.select_record_input(),
        "document": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
query	A text prompt to a vector DB. TYPE: str
document	The document returned from the vector DB. TYPE: str

RETURNS	DESCRIPTION
float	the embedding vector distance TYPE: Union[float, Tuple[float, Dict[str, str]]]

euclidean_distance

euclidean_distance(
    query: str, document: str
) -> Union[float, Tuple[float, Dict[str, str]]]

Runs L2 distance on the query and document embeddings

Example

Below is just one example. Embedders from LlamaIndex are supported: https://docs.llamaindex.ai/en/latest/module_guides/models/embeddings/

from llama_index.embeddings.openai import OpenAIEmbedding
from trulens.feedback.embeddings import Embeddings

embed_model = OpenAIEmbedding()

# Create the feedback function
from trulens.core import Metric, Selector
f_embed = feedback.Embeddings(embed_model=embed_model)
f_embed_dist = Metric(
    implementation=f_embed.euclidean_distance,
    name="Euclidean Distance",
    selectors={
        "query": Selector.select_record_input(),
        "document": Selector.select_record_output(),
    },
)

PARAMETER	DESCRIPTION
query	A text prompt to a vector DB. TYPE: str
document	The document returned from the vector DB. TYPE: str

RETURNS	DESCRIPTION
float	the embedding vector distance TYPE: Union[float, Tuple[float, Dict[str, str]]]