trulens.feedback.deterministic¶

trulens.feedback.deterministic ¶

Deterministic (non-LLM) feedback metrics.

Functions¶

conversation_repetition ¶

conversation_repetition(
    records: List[Any] | str,
    fallback_phrases: Optional[Sequence[str]] = None,
    ngram_orders: Sequence[int] = (2, 3, 4),
) -> Tuple[Optional[float], Dict[str, Any]]

Score how repetitious the assistant's replies are (deterministic).

For each assistant turn this computes four signals -- n-gram repetition within the reply, ROUGE-L overlap with the previous assistant reply, use of a known fallback phrase, and low lexical diversity -- combines them into a per-turn repetition score, and returns 1 - the worst turn's score. No LLM calls are made.

When the conversation contains no assistant turns to judge, the score is None rather than the maximum 1.0, so that "no data" stays out of any average over conversations; meta still reports num_assistant_turns as 0.

Example

from trulens.core import Metric
from trulens.feedback.deterministic import conversation_repetition

feedback = Metric(implementation=conversation_repetition)
feedback = feedback.on_conversation()

PARAMETER	DESCRIPTION
`records`	The ordered conversation records, or a transcript string. Records may be objects with `main_input`/`main_output`, dicts with `input`/`output`, or dicts with `role`/`content`. TYPE: `List[Any] \| str`
`fallback_phrases`	Phrases marking a fallback reply, matched case-insensitively. Defaults to [DEFAULT_FALLBACK_PHRASES][trulens.feedback.deterministic. DEFAULT_FALLBACK_PHRASES]. TYPE: `Optional[Sequence[str]]` DEFAULT: `None`
`ngram_orders`	n-gram orders used for the within-reply repetition signal. TYPE: `Sequence[int]` DEFAULT: `(2, 3, 4)`

RETURNS	DESCRIPTION
`Tuple[Optional[float], Dict[str, Any]]`	Tuple[Optional[float], Dict[str, Any]]: A tuple of the conversation score between 0.0 (fully repetitious) and 1.0 (no repetition) -- or `None` when there is nothing to measure -- and a meta dict with the per-turn signal breakdown, the worst turn, and the number of assistant turns.