Skip to content

trulens.apps.rl

trulens.apps.rl

Additional Dependency Required

To use this module, you must have the trulens-apps-rl package installed.

pip install trulens-apps-rl

Classes

RewardFunction

Adapts a TruLens feedback function into an RL reward signal.

Score Transformation Guidance

Selection of the transform parameter depends on your RL algorithm:

  • "2x-1" (default): Maps $[0, 1]$ feedback scores to $[-1, 1]$ reward signals. Recommended for policy gradient methods like PPO and GRPO that expect symmetric positive/negative reward signals centered at zero (positive rewards reinforce high-quality completions, negative rewards penalize poor ones).
  • "identity": Preserves original $[0, 1]$ scores unchanged. Recommended when training a Reward Model, or when using RL trainers that perform internal z-score reward normalization (e.g. TRL GRPOTrainer with reward whitening).
  • Custom callable (float) -> float: For custom reward shaping curves.
Parameters

feedback_fn: A TruLens feedback callable returning float or (float, dict). transform: Optional function or string name ("2x-1", "identity") to transform [0, 1] scores into RL scalar rewards. Defaults to "2x-1". app_name: Optional virtual app name for TruLens trajectory logging. app_version: Optional virtual app version for TruLens trajectory logging.

Functions
from_metric classmethod 
from_metric(
    metric: Any,
    *,
    transform: str | Callable[[float], float] = "2x-1",
    app_name: str | None = None,
    app_version: str | None = None
) -> RewardFunction

Create a RewardFunction directly from a TruLens Metric object or callable.

In TruLens, a :class:~trulens.core.Metric encapsulates an evaluation metric's implementation (e.g. provider.relevance or provider.groundedness), its selectors, and configuration.

Example

::

from trulens.apps.rl import RewardFunction
from trulens.core import Metric
from trulens.providers.openai import OpenAI

provider = OpenAI()
metric = Metric(
    implementation=provider.relevance,
    name="Relevance",
)
reward_fn = RewardFunction.from_metric(metric, transform="2x-1")
evaluate_sample 
evaluate_sample(
    prompt: str, completion: str, **kwargs: Any
) -> float

Evaluate a single (prompt, completion) pair and return its scalar reward.

__call__ 
__call__(
    prompts: list[str],
    completions: list[str],
    **kwargs: Any
) -> list[float]

Evaluate a batch of prompts and completions, returning a list of float rewards.

TRLRewardAdapter

Bases: RewardFunction

TRL (Transformer Reinforcement Learning) adapter wrapping TruLens metrics as TRL reward_funcs.

Supported TRL Trainers & Versions

Tested and compatible with Hugging Face TRL >= 0.7.0 (including 0.12.0+ GRPOTrainer and PPOTrainer).

TRL trainers pass decoded prompt text strings (prompts: list[str]) and completion text strings (completions: list[str]) to reward functions in the signature reward_func(prompts, completions, **kwargs) -> list[float].

Example with TRL GRPOTrainer

::

from trl import GRPOTrainer, GRPOConfig
from trulens.apps.rl import TRLRewardAdapter
from trulens.providers.openai import OpenAI

provider = OpenAI()
reward_adapter = TRLRewardAdapter(
    feedback_fn=provider.relevance,
    transform="2x-1",
)

trainer = GRPOTrainer(
    model=model,
    reward_funcs=[reward_adapter],
    train_dataset=dataset,
    args=GRPOConfig(output_dir="./results"),
)
Functions
from_metric classmethod 
from_metric(
    metric: Any,
    *,
    transform: str | Callable[[float], float] = "2x-1",
    app_name: str | None = None,
    app_version: str | None = None
) -> RewardFunction

Create a RewardFunction directly from a TruLens Metric object or callable.

In TruLens, a :class:~trulens.core.Metric encapsulates an evaluation metric's implementation (e.g. provider.relevance or provider.groundedness), its selectors, and configuration.

Example

::

from trulens.apps.rl import RewardFunction
from trulens.core import Metric
from trulens.providers.openai import OpenAI

provider = OpenAI()
metric = Metric(
    implementation=provider.relevance,
    name="Relevance",
)
reward_fn = RewardFunction.from_metric(metric, transform="2x-1")
evaluate_sample 
evaluate_sample(
    prompt: str, completion: str, **kwargs: Any
) -> float

Evaluate a single (prompt, completion) pair and return its scalar reward.

__call__ 
__call__(
    prompts: list[str],
    completions: list[str],
    **kwargs: Any
) -> list[float]

Evaluate a batch of prompts and completions, returning a list of float rewards.

Functions