trulens.apps.rl.reward¶

trulens.apps.rl.reward ¶

Reinforcement Learning (RL) reward function adapter for TruLens feedback functions.

Provides: - :class:RewardFunction: Wraps any TruLens feedback function into a scalar reward function suitable for RL fine-tuning (e.g. Hugging Face TRL PPOTrainer / GRPOTrainer). - :class:TRLRewardAdapter: High-level adapter specifically designed for TRL reward_funcs signature.

Classes¶

RewardFunction ¶

Adapts a TruLens feedback function into an RL reward signal.

Score Transformation Guidance¶

Selection of the transform parameter depends on your RL algorithm:

"2x-1" (default): Maps $[0, 1]$ feedback scores to $[-1, 1]$ reward signals. Recommended for policy gradient methods like PPO and GRPO that expect symmetric positive/negative reward signals centered at zero (positive rewards reinforce high-quality completions, negative rewards penalize poor ones).
"identity": Preserves original $[0, 1]$ scores unchanged. Recommended when training a Reward Model, or when using RL trainers that perform internal z-score reward normalization (e.g. TRL GRPOTrainer with reward whitening).
Custom callable (float) -> float: For custom reward shaping curves.

Parameters¶

feedback_fn: A TruLens feedback callable returning float or (float, dict). transform: Optional function or string name ("2x-1", "identity") to transform [0, 1] scores into RL scalar rewards. Defaults to "2x-1". app_name: Optional virtual app name for TruLens trajectory logging. app_version: Optional virtual app version for TruLens trajectory logging.

Functions¶

from_metric `classmethod` ¶

from_metric(
    metric: Any,
    *,
    transform: str | Callable[[float], float] = "2x-1",
    app_name: str | None = None,
    app_version: str | None = None
) -> RewardFunction

Create a RewardFunction directly from a TruLens Metric object or callable.

In TruLens, a :class:~trulens.core.Metric encapsulates an evaluation metric's implementation (e.g. provider.relevance or provider.groundedness), its selectors, and configuration.

Example¶

::

from trulens.apps.rl import RewardFunction
from trulens.core import Metric
from trulens.providers.openai import OpenAI

provider = OpenAI()
metric = Metric(
    implementation=provider.relevance,
    name="Relevance",
)
reward_fn = RewardFunction.from_metric(metric, transform="2x-1")

evaluate_sample ¶

evaluate_sample(
    prompt: str, completion: str, **kwargs: Any
) -> float

Evaluate a single (prompt, completion) pair and return its scalar reward.

call ¶

__call__(
    prompts: list[str],
    completions: list[str],
    **kwargs: Any
) -> list[float]

Evaluate a batch of prompts and completions, returning a list of float rewards.

TRLRewardAdapter ¶

Bases: RewardFunction

TRL (Transformer Reinforcement Learning) adapter wrapping TruLens metrics as TRL reward_funcs.

Supported TRL Trainers & Versions¶

Tested and compatible with Hugging Face TRL >= 0.7.0 (including 0.12.0+ GRPOTrainer and PPOTrainer).

TRL trainers pass decoded prompt text strings (prompts: list[str]) and completion text strings (completions: list[str]) to reward functions in the signature reward_func(prompts, completions, **kwargs) -> list[float].

Example with TRL GRPOTrainer¶

::

from trl import GRPOTrainer, GRPOConfig
from trulens.apps.rl import TRLRewardAdapter
from trulens.providers.openai import OpenAI

provider = OpenAI()
reward_adapter = TRLRewardAdapter(
    feedback_fn=provider.relevance,
    transform="2x-1",
)

trainer = GRPOTrainer(
    model=model,
    reward_funcs=[reward_adapter],
    train_dataset=dataset,
    args=GRPOConfig(output_dir="./results"),
)

Functions¶

from_metric `classmethod` ¶

from_metric(
    metric: Any,
    *,
    transform: str | Callable[[float], float] = "2x-1",
    app_name: str | None = None,
    app_version: str | None = None
) -> RewardFunction

Create a RewardFunction directly from a TruLens Metric object or callable.

In TruLens, a :class:~trulens.core.Metric encapsulates an evaluation metric's implementation (e.g. provider.relevance or provider.groundedness), its selectors, and configuration.

Example¶

::

from trulens.apps.rl import RewardFunction
from trulens.core import Metric
from trulens.providers.openai import OpenAI

provider = OpenAI()
metric = Metric(
    implementation=provider.relevance,
    name="Relevance",
)
reward_fn = RewardFunction.from_metric(metric, transform="2x-1")

evaluate_sample ¶

evaluate_sample(
    prompt: str, completion: str, **kwargs: Any
) -> float

Evaluate a single (prompt, completion) pair and return its scalar reward.

call ¶

__call__(
    prompts: list[str],
    completions: list[str],
    **kwargs: Any
) -> list[float]

Evaluate a batch of prompts and completions, returning a list of float rewards.

Functions¶

transform_2x_minus_1 ¶

transform_2x_minus_1(score: float) -> float

Transform score in [0, 1] to reward signal in [-1, 1].

transform_identity ¶

transform_identity(score: float) -> float

Identity transform (returns score unchanged in [0, 1]).

trulens.apps.rl.reward¶