trulens.core.feedback.feedback¶

trulens.core.feedback.feedback ¶

Attributes¶

ImpCallable `module-attribute` ¶

ImpCallable = Callable[
    [A], Union[float, Tuple[float, Dict[str, Any]]]
]

Signature of feedback implementations.

Those take in any number of arguments and return either a single float or a float and a dictionary (of metadata).

AggCallable `module-attribute` ¶

AggCallable = Callable[
    [Union[Iterable[float], Iterable[Tuple[float, float]]]],
    float,
]

Signature of aggregation functions.

Classes¶

SkipEval ¶

Bases: Exception

Raised when evaluating a feedback function implementation to skip it so it is not aggregated with other non-skipped results.

PARAMETER	DESCRIPTION
`reason`	Optional reason for why this evaluation was skipped. TYPE: `Optional[str]` DEFAULT: `None`
`feedback`	The Feedback instance this run corresponds to. TYPE: `Optional[Feedback]` DEFAULT: `None`
`ins`	The arguments to this run. TYPE: `Optional[Dict[str, Any]]` DEFAULT: `None`

InvalidSelector ¶

Bases: Exception

Raised when a selector names something that is missing in a record/app.

Feedback ¶

Bases: FeedbackDefinition

Feedback function container.

Typical usage is to specify a feedback implementation function from a Provider and the mapping of selectors describing how to construct the arguments to the implementation:

Example

from trulens.core import Feedback
from trulens.providers.huggingface import Huggingface
hugs = Huggingface()

# Create a feedback function from a provider:
feedback = Feedback(
    hugs.language_match # the implementation
).on_input_output() # selectors shorthand

Attributes¶

tru_class_info `instance-attribute` ¶

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

implementation `class-attribute` `instance-attribute` ¶

implementation: Optional[Union[Function, Method]] = None

Implementation serialization.

aggregator `class-attribute` `instance-attribute` ¶

aggregator: Optional[Union[Function, Method]] = None

Aggregator method serialization.

combinations `class-attribute` `instance-attribute` ¶

combinations: Optional[FeedbackCombinations] = PRODUCT

Mode of combining selected values to produce arguments to each feedback function call.

feedback_definition_id `instance-attribute` ¶

feedback_definition_id: FeedbackDefinitionID = (
    feedback_definition_id
)

Id, if not given, uniquely determined from content.

if_exists `class-attribute` `instance-attribute` ¶

if_exists: Optional[Lens] = None

Only execute the feedback function if the following selector names something that exists in a record/app.

Can use this to evaluate conditionally on presence of some calls, for example. Feedbacks skipped this way will have a status of FeedbackResultStatus.SKIPPED.

if_missing `class-attribute` `instance-attribute` ¶

if_missing: FeedbackOnMissingParameters = ERROR

How to handle missing parameters in feedback function calls.

run_location `instance-attribute` ¶

run_location: Optional[FeedbackRunLocation]

Where the feedback evaluation takes place (e.g. locally, at a Snowflake server, etc).

selectors `instance-attribute` ¶

selectors: Dict[str, Lens]

Selectors; pointers into Records of where to get arguments for imp.

supplied_name `class-attribute` `instance-attribute` ¶

supplied_name: Optional[str] = None

An optional name. Only will affect displayed tables.

higher_is_better `class-attribute` `instance-attribute` ¶

higher_is_better: Optional[bool] = None

Feedback result magnitude interpretation.

imp `class-attribute` `instance-attribute` ¶

imp: Optional[ImpCallable] = imp

Implementation callable.

A serialized version is stored at FeedbackDefinition.implementation.

agg `class-attribute` `instance-attribute` ¶

agg: Optional[AggCallable] = agg

Aggregator method for feedback functions that produce more than one result.

A serialized version is stored at FeedbackDefinition.aggregator.

sig `property` ¶

sig: Signature

Signature of the feedback function implementation.

name `property` ¶

name: str

Name of the feedback function.

Derived from the name of the function implementing it if no supplied name provided.

Functions¶

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load `staticmethod` ¶

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate `classmethod` ¶

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

on_input_output ¶

on_input_output() -> Feedback

Specifies that the feedback implementation arguments are to be the main app input and output in that order.

Returns a new Feedback object with the specification.

on_default ¶

on_default() -> Feedback

Specifies that one argument feedbacks should be evaluated on the main app output and two argument feedbacks should be evaluates on main input and main output in that order.

Returns a new Feedback object with this specification.

evaluate_deferred `staticmethod` ¶

evaluate_deferred(
    session: TruSession,
    limit: Optional[int] = None,
    shuffle: bool = False,
    run_location: Optional[FeedbackRunLocation] = None,
) -> List[Tuple[Series, Future[FeedbackResult]]]

Evaluates feedback functions that were specified to be deferred.

Returns a list of tuples with the DB row containing the Feedback and initial FeedbackResult as well as the Future which will contain the actual result.

PARAMETER	DESCRIPTION
`limit`	The maximum number of evals to start. TYPE: `Optional[int]` DEFAULT: `None`
`shuffle`	Shuffle the order of the feedbacks to evaluate. TYPE: `bool` DEFAULT: `False`
`run_location`	Only run feedback functions with this run_location. TYPE: `Optional[FeedbackRunLocation]` DEFAULT: `None`

Constants that govern behavior:

TruSession.RETRY_RUNNING_SECONDS: How long to time before restarting a feedback that was started but never failed (or failed without recording that fact).
TruSession.RETRY_FAILED_SECONDS: How long to wait to retry a failed feedback.

aggregate ¶

aggregate(
    func: Optional[AggCallable] = None,
    combinations: Optional[FeedbackCombinations] = None,
) -> Feedback

Specify the aggregation function in case the selectors for this feedback generate more than one value for implementation argument(s). Can also specify the method of producing combinations of values in such cases.

Returns a new Feedback object with the given aggregation function and/or the given combination mode.

on_prompt ¶

on_prompt(arg: Optional[str] = None) -> Feedback

Create a variant of self that will take in the main app input or "prompt" as input, sending it as an argument arg to implementation.

on_response ¶

on_response(arg: Optional[str] = None) -> Feedback

Create a variant of self that will take in the main app output or "response" as input, sending it as an argument arg to implementation.

on ¶

on(*args, **kwargs) -> Feedback

Create a variant of self with the same implementation but the given selectors. Those provided positionally get their implementation argument name guessed and those provided as kwargs get their name from the kwargs key.

check_selectors ¶

check_selectors(
    app: Union[AppDefinition, JSON],
    record: Record,
    source_data: Optional[Dict[str, Any]] = None,
    warning: bool = False,
) -> bool

Check that the selectors are valid for the given app and record.

PARAMETER	DESCRIPTION
`app`	The app that produced the record. TYPE: `Union[AppDefinition, JSON]`
`record`	The record that the feedback will run on. This can be a mostly empty record for checking ahead of producing one. The utility method App.dummy_record is built for this purpose. TYPE: `Record`
`source_data`	Additional data to select from when extracting feedback function arguments. TYPE: `Optional[Dict[str, Any]]` DEFAULT: `None`
`warning`	Issue a warning instead of raising an error if a selector is invalid. As some parts of a Record cannot be known ahead of producing it, it may be necessary to not raise exception here and only issue a warning. TYPE: `bool` DEFAULT: `False`

RETURNS	DESCRIPTION
`bool`	True if the selectors are valid. False if not (if warning is set).

RAISES	DESCRIPTION
`ValueError`	If a selector is invalid and warning is not set.

run ¶

run(
    app: Optional[Union[AppDefinition, JSON]] = None,
    record: Optional[Record] = None,
    source_data: Optional[Dict] = None,
    **kwargs: Dict[str, Any]
) -> FeedbackResult

Run the feedback function on the given record. The app that produced the record is also required to determine input/output argument names.

PARAMETER	DESCRIPTION
`app`	The app that produced the record. This can be AppDefinition or a jsonized AppDefinition. It will be jsonized if it is not already. TYPE: `Optional[Union[AppDefinition, JSON]]` DEFAULT: `None`
`record`	The record to evaluate the feedback on. TYPE: `Optional[Record]` DEFAULT: `None`
`source_data`	Additional data to select from when extracting feedback function arguments. TYPE: `Optional[Dict]` DEFAULT: `None`
`**kwargs`	Any additional keyword arguments are used to set or override selected feedback function inputs. TYPE: `Dict[str, Any]` DEFAULT: `{}`

RETURNS	DESCRIPTION
`FeedbackResult`	A FeedbackResult object with the result of the feedback function.

extract_selection ¶

extract_selection(
    app: Optional[Union[AppDefinition, JSON]] = None,
    record: Optional[Record] = None,
    source_data: Optional[Dict] = None,
) -> Iterable[Dict[str, Any]]

Given the app that produced the given record, extract from record the values that will be sent as arguments to the implementation as specified by self.selectors. Additional data to select from can be provided in source_data. All args are optional. If a Record is specified, its calls are laid out as app (see layout_calls_as_app).

SnowflakeFeedback ¶

Bases: Feedback

Similar to the parent class Feedback except this ensures the feedback is run only on the Snowflake server.

Attributes¶

tru_class_info `instance-attribute` ¶

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

implementation `class-attribute` `instance-attribute` ¶

implementation: Optional[Union[Function, Method]] = None

Implementation serialization.

aggregator `class-attribute` `instance-attribute` ¶

aggregator: Optional[Union[Function, Method]] = None

Aggregator method serialization.

combinations `class-attribute` `instance-attribute` ¶

combinations: Optional[FeedbackCombinations] = PRODUCT

Mode of combining selected values to produce arguments to each feedback function call.

feedback_definition_id `instance-attribute` ¶

feedback_definition_id: FeedbackDefinitionID = (
    feedback_definition_id
)

Id, if not given, uniquely determined from content.

if_exists `class-attribute` `instance-attribute` ¶

if_exists: Optional[Lens] = None

Only execute the feedback function if the following selector names something that exists in a record/app.

Can use this to evaluate conditionally on presence of some calls, for example. Feedbacks skipped this way will have a status of FeedbackResultStatus.SKIPPED.

if_missing `class-attribute` `instance-attribute` ¶

if_missing: FeedbackOnMissingParameters = ERROR

How to handle missing parameters in feedback function calls.

selectors `instance-attribute` ¶

selectors: Dict[str, Lens]

Selectors; pointers into Records of where to get arguments for imp.

supplied_name `class-attribute` `instance-attribute` ¶

supplied_name: Optional[str] = None

An optional name. Only will affect displayed tables.

higher_is_better `class-attribute` `instance-attribute` ¶

higher_is_better: Optional[bool] = None

Feedback result magnitude interpretation.

name `property` ¶

name: str

Name of the feedback function.

Derived from the name of the function implementing it if no supplied name provided.

imp `class-attribute` `instance-attribute` ¶

imp: Optional[ImpCallable] = imp

Implementation callable.

A serialized version is stored at FeedbackDefinition.implementation.

agg `class-attribute` `instance-attribute` ¶

agg: Optional[AggCallable] = agg

Aggregator method for feedback functions that produce more than one result.

A serialized version is stored at FeedbackDefinition.aggregator.

sig `property` ¶

sig: Signature

Signature of the feedback function implementation.

Functions¶

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load `staticmethod` ¶

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate `classmethod` ¶

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

on_input_output ¶

on_input_output() -> Feedback

Specifies that the feedback implementation arguments are to be the main app input and output in that order.

Returns a new Feedback object with the specification.

on_default ¶

on_default() -> Feedback

Specifies that one argument feedbacks should be evaluated on the main app output and two argument feedbacks should be evaluates on main input and main output in that order.

Returns a new Feedback object with this specification.

evaluate_deferred `staticmethod` ¶

evaluate_deferred(
    session: TruSession,
    limit: Optional[int] = None,
    shuffle: bool = False,
    run_location: Optional[FeedbackRunLocation] = None,
) -> List[Tuple[Series, Future[FeedbackResult]]]

Evaluates feedback functions that were specified to be deferred.

Returns a list of tuples with the DB row containing the Feedback and initial FeedbackResult as well as the Future which will contain the actual result.

PARAMETER	DESCRIPTION
`limit`	The maximum number of evals to start. TYPE: `Optional[int]` DEFAULT: `None`
`shuffle`	Shuffle the order of the feedbacks to evaluate. TYPE: `bool` DEFAULT: `False`
`run_location`	Only run feedback functions with this run_location. TYPE: `Optional[FeedbackRunLocation]` DEFAULT: `None`

Constants that govern behavior:

TruSession.RETRY_RUNNING_SECONDS: How long to time before restarting a feedback that was started but never failed (or failed without recording that fact).
TruSession.RETRY_FAILED_SECONDS: How long to wait to retry a failed feedback.

aggregate ¶

aggregate(
    func: Optional[AggCallable] = None,
    combinations: Optional[FeedbackCombinations] = None,
) -> Feedback

Specify the aggregation function in case the selectors for this feedback generate more than one value for implementation argument(s). Can also specify the method of producing combinations of values in such cases.

Returns a new Feedback object with the given aggregation function and/or the given combination mode.

on_prompt ¶

on_prompt(arg: Optional[str] = None) -> Feedback

Create a variant of self that will take in the main app input or "prompt" as input, sending it as an argument arg to implementation.

on_response ¶

on_response(arg: Optional[str] = None) -> Feedback

Create a variant of self that will take in the main app output or "response" as input, sending it as an argument arg to implementation.

on ¶

on(*args, **kwargs) -> Feedback

Create a variant of self with the same implementation but the given selectors. Those provided positionally get their implementation argument name guessed and those provided as kwargs get their name from the kwargs key.

check_selectors ¶

check_selectors(
    app: Union[AppDefinition, JSON],
    record: Record,
    source_data: Optional[Dict[str, Any]] = None,
    warning: bool = False,
) -> bool

Check that the selectors are valid for the given app and record.

PARAMETER	DESCRIPTION
`app`	The app that produced the record. TYPE: `Union[AppDefinition, JSON]`
`record`	The record that the feedback will run on. This can be a mostly empty record for checking ahead of producing one. The utility method App.dummy_record is built for this purpose. TYPE: `Record`
`source_data`	Additional data to select from when extracting feedback function arguments. TYPE: `Optional[Dict[str, Any]]` DEFAULT: `None`
`warning`	Issue a warning instead of raising an error if a selector is invalid. As some parts of a Record cannot be known ahead of producing it, it may be necessary to not raise exception here and only issue a warning. TYPE: `bool` DEFAULT: `False`

RETURNS	DESCRIPTION
`bool`	True if the selectors are valid. False if not (if warning is set).

RAISES	DESCRIPTION
`ValueError`	If a selector is invalid and warning is not set.

run ¶

run(
    app: Optional[Union[AppDefinition, JSON]] = None,
    record: Optional[Record] = None,
    source_data: Optional[Dict] = None,
    **kwargs: Dict[str, Any]
) -> FeedbackResult

Run the feedback function on the given record. The app that produced the record is also required to determine input/output argument names.

PARAMETER	DESCRIPTION
`app`	The app that produced the record. This can be AppDefinition or a jsonized AppDefinition. It will be jsonized if it is not already. TYPE: `Optional[Union[AppDefinition, JSON]]` DEFAULT: `None`
`record`	The record to evaluate the feedback on. TYPE: `Optional[Record]` DEFAULT: `None`
`source_data`	Additional data to select from when extracting feedback function arguments. TYPE: `Optional[Dict]` DEFAULT: `None`
`**kwargs`	Any additional keyword arguments are used to set or override selected feedback function inputs. TYPE: `Dict[str, Any]` DEFAULT: `{}`

RETURNS	DESCRIPTION
`FeedbackResult`	A FeedbackResult object with the result of the feedback function.

extract_selection ¶

extract_selection(
    app: Optional[Union[AppDefinition, JSON]] = None,
    record: Optional[Record] = None,
    source_data: Optional[Dict] = None,
) -> Iterable[Dict[str, Any]]

Given the app that produced the given record, extract from record the values that will be sent as arguments to the implementation as specified by self.selectors. Additional data to select from can be provided in source_data. All args are optional. If a Record is specified, its calls are laid out as app (see layout_calls_as_app).

trulens.core.feedback.feedback¶