trulens.feedback.optimize¶

trulens.feedback.optimize ¶

Few-shot example optimizer for TruLens feedback functions.

This module provides :class:FewShotOptimizer, a utility for selecting the best-performing subset of few-shot examples to include in an LLM judge prompt.

Motivation¶

Feedback functions in TruLens accept an examples parameter that injects demonstration examples into the judge's system prompt. Choosing which examples to include has a large effect on scoring quality, but there is currently no principled way to pick them. FewShotOptimizer fills this gap by scoring every candidate example against a labeled dataset and returning the subset that maximises agreement with ground-truth scores.

Typical usage¶

::

from trulens.providers.openai import OpenAI
from trulens.feedback.optimize import FewShotOptimizer

provider = OpenAI()

# A pool of candidate demonstrations: each entry is a
# (feedback_kwargs, ground_truth_score) pair.
candidates = [
    ({"input": "What is 2+2?", "output": "4"},          1.0),
    ({"input": "What is the capital?", "output": "Paris"}, 0.9),
    ({"input": "Who wrote Hamlet?", "output": "Einstein"}, 0.1),
    # … more examples …
]

# A separate held-out dataset used to *evaluate* which examples help most.
eval_dataset = [
    ({"input": "Explain gravity.", "output": "A force."}, 0.8),
    # …
]

optimizer = FewShotOptimizer(
    feedback_fn=provider.relevance,
    candidates=candidates,
    eval_dataset=eval_dataset,
    n_examples=3,
)
best_examples = optimizer.optimize()

# Use the optimized examples with your feedback function.
provider.relevance(
    input="…",
    output="…",
    examples=optimizer.format_examples(best_examples),
)

Classes¶

OptimizeResult `dataclass` ¶

Outcome of a :meth:FewShotOptimizer.optimize run.

Attributes¶

best_examples: The subset of candidates selected by the optimizer, each paired with its ground-truth score. correlation: Evaluation score (e.g. Pearson correlation or selected metric score) achieved on eval_dataset when using best_examples. Higher is better. None if fewer than two eval samples were available. candidate_scores: Mapping from candidate index → metric score achieved when that candidate was included in the prompt. Useful for debugging. metric_name: Name of the metric used for optimization (e.g. "pearson", "f1"). metric_score: The metric score achieved by best_examples.

FewShotOptimizer ¶

Select the best few-shot examples for a TruLens feedback function.

The optimizer works by:

Iterating over candidates one at a time in parallel rounds.
For each candidate, calling feedback_fn on every sample in eval_dataset with that candidate injected as a few-shot example.
Computing the target metric (e.g., Pearson correlation, F1, precision, recall, Cohen's kappa) between predicted and ground-truth scores.
Greedily selecting the n_examples candidates with the highest metric improvement (greedy forward selection).

Parameters¶

feedback_fn: A callable that accepts the keyword arguments defined in candidates plus an optional examples: str keyword argument. It must return a float in [0, 1]. Must be thread-safe; called concurrently when max_workers > 1. Typically a bound method on a :class:trulens.feedback.LLMProvider subclass, e.g. provider.relevance. candidates: Pool of demonstration examples to select from. Each entry is a (feedback_kwargs, ground_truth_score) pair where ground_truth_score is a float in [0, 1]. eval_dataset: Held-out labeled examples used to measure how well a candidate set helps the judge. Should be disjoint from candidates to avoid overfitting. n_examples: Maximum number of examples to include in the final prompt. Defaults to 3. format_sep: Separator inserted between formatted examples when building the examples string passed to feedback_fn. Defaults to "\n\n". metric: Evaluation metric to optimize. Supported metrics: "pearson", "spearman", "precision", "recall", "f1", "cohens_kappa", "accuracy", "mae". Defaults to "pearson". metric_threshold: Threshold used for binarizing scores when computing classification metrics (precision, recall, f1, cohens_kappa, accuracy). Defaults to 0.5. max_workers: Maximum number of thread workers used to evaluate candidate sets in parallel. Defaults to None (utilizes ThreadPoolExecutor's default).

Functions¶

optimize ¶

optimize() -> OptimizeResult

Run greedy forward selection and return the best example subset.

Returns¶

OptimizeResult Contains the selected examples, overall evaluation score, and per-candidate scores.

Raises¶

RuntimeError If feedback_fn raises an exception for every candidate on the first eval sample (likely a misconfigured provider).

format_examples ¶

format_examples(examples: list[LabeledExample]) -> str

Serialise a list of labeled examples into the string format expected by feedback_fn's examples parameter.

Each example is rendered as a bullet showing the input kwargs and the expected score, separated by :attr:format_sep.

Parameters¶

examples: Subset of labeled examples to format, typically the output of :meth:optimize.

Returns¶

str A human-readable string ready to be passed as feedback_fn(..., examples=<return_value>).

trulens.feedback.optimize¶