Skip to content

Feedback

trulens_eval.feedback.feedback.ImpCallable module-attribute

ImpCallable = Callable[
    [A], Union[float, Tuple[float, Dict[str, Any]]]
]

Signature of feedback implementations.

Those take in any number of arguments and return either a single float or a float and a dictionary (of metadata).

trulens_eval.feedback.feedback.AggCallable module-attribute

AggCallable = Callable[[Iterable[float]], float]

Signature of aggregation functions.

trulens_eval.feedback.feedback.Feedback

Bases: FeedbackDefinition

Feedback function container.

Typical usage is to specify a feedback implementation function from a Provider and the mapping of selectors describing how to construct the arguments to the implementation:

Example
from trulens_eval import Feedback
from trulens_eval import Huggingface
hugs = Huggingface()

# Create a feedback function from a provider:
feedback = Feedback(
    hugs.language_match # the implementation
).on_input_output() # selectors shorthand

Attributes

imp class-attribute instance-attribute

Implementation callable. A serialized version is stored at FeedbackDefinition.implementation.

agg class-attribute instance-attribute

Aggregator method for feedback functions that produce more than one result. A serialized version is stored at FeedbackDefinition.aggregator.

name property

name: str

Name of the feedback function.

Derived from the name of the function implementing it if no supplied name provided.

Functions

on_input_output

on_input_output() -> Feedback

Specifies that the feedback implementation arguments are to be the main app input and output in that order.

Returns a new Feedback object with the specification.

on_default

on_default() -> Feedback

Specifies that one argument feedbacks should be evaluated on the main app output and two argument feedbacks should be evaluates on main input and main output in that order.

Returns a new Feedback object with this specification.

evaluate_deferred staticmethod

evaluate_deferred(
    tru: Tru,
    limit: Optional[int] = None,
    shuffle: bool = False,
) -> List[Tuple[Series, Future[FeedbackResult]]]

Evaluates feedback functions that were specified to be deferred. Returns a list of tuples with the DB row containing the Feedback and initial FeedbackResult as well as the Future which will contain the actual result.

  • limit: Optional[int] -- indicates the maximum number of evals to start.

  • shuffle: bool -- shuffles the order of the feedbacks to evaluate.

Constants that govern behaviour:

  • Tru.RETRY_RUNNING_SECONDS: How long to time before restarting a feedback that was started but never failed (or failed without recording that fact).

  • Tru.RETRY_FAILED_SECONDS: How long to wait to retry a failed feedback.

aggregate

aggregate(func: AggCallable) -> Feedback

Specify the aggregation function in case the selectors for this feedback generate more than one value for implementation argument(s).

Returns a new Feedback object with the given aggregation function.

on_prompt

on_prompt(arg: Optional[str] = None) -> Feedback

Create a variant of self that will take in the main app input or "prompt" as input, sending it as an argument arg to implementation.

on_response

on_response(arg: Optional[str] = None) -> Feedback

Create a variant of self that will take in the main app output or "response" as input, sending it as an argument arg to implementation.

on

on(*args, **kwargs) -> Feedback

Create a variant of self with the same implementation but the given selectors. Those provided positionally get their implementation argument name guessed and those provided as kwargs get their name from the kwargs key.

run

run(
    app: Union[AppDefinition, JSON], record: Record
) -> FeedbackResult

Run the feedback function on the given record. The app that produced the record is also required to determine input/output argument names.

Might not have a AppDefinitionhere but only the serialized app_json .

extract_selection

extract_selection(
    app: Union[AppDefinition, JSON], record: Record
) -> Iterable[Dict[str, Any]]

Given the app that produced the given record, extract from record the values that will be sent as arguments to the implementation as specified by self.selectors.

trulens_eval.schema.FeedbackDefinition

Bases: WithClassInfo, SerialModel, Hashable

Serialized parts of a feedback function.

The non-serialized parts are in the Feedback class.

Attributes

implementation class-attribute instance-attribute

implementation: Optional[Union[Function, Method]] = None

Implementation serialization.

aggregator class-attribute instance-attribute

aggregator: Optional[Union[Function, Method]] = None

Aggregator method serialization.

selectors instance-attribute

selectors: Dict[str, Lens]

Selectors; pointers into Records of where to get arguments for imp.

supplied_name class-attribute instance-attribute

supplied_name: Optional[str] = None

An optional name. Only will affect displayed tables.

higher_is_better class-attribute instance-attribute

higher_is_better: Optional[bool] = None

Feedback result magnitude interpretation.

feedback_definition_id instance-attribute

feedback_definition_id: FeedbackDefinitionID = (
    feedback_definition_id
)

Id, if not given, uniquely determined from content.

name property

name: str

Name of the feedback function.

Derived from the name of the serialized implementation function if name was not provided.