trulens.feedback.optimize¶

trulens.feedback.optimize ¶

Few-shot example optimizer for TruLens feedback functions.

This module provides :class:FewShotOptimizer, a utility for selecting the best-performing subset of few-shot examples to include in an LLM judge prompt.

Motivation¶

Feedback functions in TruLens accept an examples parameter that injects demonstration examples into the judge's system prompt. Choosing which examples to include has a large effect on scoring quality, but there is currently no principled way to pick them. FewShotOptimizer fills this gap by scoring every candidate example against a labeled dataset and returning the subset that maximises agreement with ground-truth scores.

Typical usage¶

::

from trulens.providers.openai import OpenAI
from trulens.feedback.optimize import FewShotOptimizer

provider = OpenAI()

# Candidate demonstrations use the judge's prompt score scale.
candidates = [
    ({"prompt": "What is 2+2?", "response": "4"}, 3),
    ({"prompt": "What is the capital?", "response": "Paris"}, 3),
    ({"prompt": "Who wrote Hamlet?", "response": "Einstein"}, 0),
    # … more examples …
]

# A separate held-out dataset used to *evaluate* which examples help most.
eval_dataset = [
    ({"prompt": "Explain gravity.", "response": "A force."}, 0.8),
    # …
]

optimizer = FewShotOptimizer(
    feedback_fn=provider.relevance,
    candidates=candidates,
    eval_dataset=eval_dataset,
    n_examples=3,
    examples_format="structured",
)
result = optimizer.optimize()

# Standard TruLens provider methods accept the structured list directly.
provider.relevance(
    prompt="…",
    response="…",
    examples=result.best_examples,
)

Classes¶

OptimizeResult `dataclass` ¶

Outcome of a :meth:FewShotOptimizer.optimize run.

Attributes¶

best_examples: The subset of candidates selected by the optimizer, each paired with its ground-truth score. correlation: Evaluation score (e.g. Pearson correlation or selected metric score) achieved on eval_dataset when using best_examples. Higher is better. None if fewer than two eval samples were available. candidate_scores: Mapping from candidate index → metric score achieved when that candidate was included in the prompt. Useful for debugging. metric_name: Name of the metric used for optimization (e.g. "pearson", "f1"). metric_score: The metric score achieved by best_examples.

FewShotOptimizer ¶

Select the best few-shot examples for a TruLens feedback function.

The optimizer works by:

Iterating over candidates one at a time in parallel rounds.
For each candidate, calling feedback_fn on every sample in eval_dataset with that candidate injected as a few-shot example.
Computing the target metric (e.g., Pearson correlation, F1, precision, recall, Cohen's kappa) between predicted and ground-truth scores.
Greedily selecting the n_examples candidates with the highest metric improvement (greedy forward selection).

Parameters¶

feedback_fn: A callable that accepts the keyword arguments defined in candidates plus an optional examples keyword argument. It must return a float in [0, 1]. Must be thread-safe; called concurrently when max_workers > 1. Typically a bound method on a :class:trulens.feedback.LLMProvider subclass, e.g. provider.relevance. candidates: Pool of demonstration examples to select from. Each entry is a (feedback_kwargs, score) pair. With examples_format="structured", the score must be an integer on the feedback function's prompt scale. eval_dataset: Held-out labeled examples used to measure how well a candidate set helps the judge. Should be disjoint from candidates to avoid overfitting. n_examples: Maximum number of examples to include in the final prompt. Defaults to 3. format_sep: Separator inserted between examples returned by :meth:format_examples. Standard TruLens providers use the structured examples directly. Defaults to "\n\n". examples_format: "text" preserves the original behavior and passes a formatted string to custom feedback callables. "structured" passes the list of (feedback_kwargs, score) tuples expected by standard TruLens providers. Defaults to "text". metric: Evaluation metric to optimize. Supported metrics: "pearson", "spearman", "precision", "recall", "f1", "cohens_kappa", "accuracy", "mae". Defaults to "pearson". metric_threshold: Threshold used for binarizing scores when computing classification metrics (precision, recall, f1, cohens_kappa, accuracy). Defaults to 0.5. max_workers: Maximum number of thread workers used to evaluate candidate sets in parallel. Defaults to None (utilizes ThreadPoolExecutor's default).

Functions¶

optimize ¶

optimize() -> OptimizeResult

Run greedy forward selection and return the best example subset.

Returns¶

OptimizeResult Contains the selected examples, overall evaluation score, and per-candidate scores.

Raises¶

RuntimeError If feedback_fn raises an exception for every candidate on the first eval sample (likely a misconfigured provider).

format_examples ¶

format_examples(examples: list[LabeledExample]) -> str

Serialize examples for custom callables that accept a string.

Each example is rendered as a bullet showing the input kwargs and the expected score, separated by :attr:format_sep.

Parameters¶

examples: Subset of labeled examples to format, typically the output of :meth:optimize.

Returns¶

str A human-readable representation. Standard TruLens provider methods should receive the structured examples instead.

trulens.feedback.optimize¶