Feedback¶
Feedback functions are stored as instances of Feedback which itself extends FeedbackDefinition. The definition parent contains serializable fields while the non-definition subclass adds non-serializable instantiations.
Bases: FeedbackDefinition
Feedback function container.
Typical usage is to specify a feedback implementation function from a Provider and the mapping of selectors describing how to construct the arguments to the implementation:
Example
from trulens_eval import Feedback
from trulens_eval import Huggingface
hugs = Huggingface()
# Create a feedback function from a provider:
feedback = Feedback(
hugs.language_match # the implementation
).on_input_output() # selectors shorthand
Attributes¶
class-attribute
instance-attribute
¶
imp: Optional[ImpCallable] = imp
Implementation callable.
A serialized version is stored at FeedbackDefinition.implementation.
class-attribute
instance-attribute
¶
agg: Optional[AggCallable] = agg
Aggregator method for feedback functions that produce more than one result.
A serialized version is stored at FeedbackDefinition.aggregator.
property
¶
name: str
Name of the feedback function.
Derived from the name of the function implementing it if no supplied name provided.
Functions¶
on_input_output() -> Feedback
Specifies that the feedback implementation arguments are to be the main app input and output in that order.
Returns a new Feedback object with the specification.
on_default() -> Feedback
Specifies that one argument feedbacks should be evaluated on the main app output and two argument feedbacks should be evaluates on main input and main output in that order.
Returns a new Feedback object with this specification.
staticmethod
¶
evaluate_deferred(tru: Tru, limit: Optional[int] = None, shuffle: bool = False) -> List[Tuple[Series, Future[FeedbackResult]]]
Evaluates feedback functions that were specified to be deferred.
Returns a list of tuples with the DB row containing the Feedback and initial FeedbackResult as well as the Future which will contain the actual result.
| PARAMETER | DESCRIPTION |
|---|---|
limit |
The maximum number of evals to start. |
shuffle |
Shuffle the order of the feedbacks to evaluate.
TYPE:
|
Constants that govern behaviour:
-
Tru.RETRY_RUNNING_SECONDS: How long to time before restarting a feedback that was started but never failed (or failed without recording that fact).
-
Tru.RETRY_FAILED_SECONDS: How long to wait to retry a failed feedback.
aggregate(func: Optional[AggCallable] = None, combinations: Optional[FeedbackCombinations] = None) -> Feedback
Specify the aggregation function in case the selectors for this feedback generate more than one value for implementation argument(s). Can also specify the method of producing combinations of values in such cases.
Returns a new Feedback object with the given aggregation function and/or the given combination mode.
Create a variant of self that will take in the main app input or
"prompt" as input, sending it as an argument arg to implementation.
Create a variant of self that will take in the main app output or
"response" as input, sending it as an argument arg to implementation.
on(*args, **kwargs) -> Feedback
Create a variant of self with the same implementation but the given
selectors. Those provided positionally get their implementation argument
name guessed and those provided as kwargs get their name from the kwargs
key.
check_selectors(app: Union[AppDefinition, JSON], record: Record, source_data: Optional[Dict[str, Any]] = None, warning: bool = False) -> bool
Check that the selectors are valid for the given app and record.
| PARAMETER | DESCRIPTION |
|---|---|
app |
The app that produced the record.
TYPE:
|
record |
The record that the feedback will run on. This can be a mostly empty record for checking ahead of producing one. The utility method App.dummy_record is built for this prupose.
TYPE:
|
source_data |
Additional data to select from when extracting feedback function arguments. |
warning |
Issue a warning instead of raising an error if a selector is invalid. As some parts of a Record cannot be known ahead of producing it, it may be necessary to not raise exception here and only issue a warning.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the selectors are valid. False if not (if warning is set). |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If a selector is invalid and warning is not set. |
run(app: Optional[Union[AppDefinition, JSON]] = None, record: Optional[Record] = None, source_data: Optional[Dict] = None, **kwargs: Dict[str, Any]) -> FeedbackResult
Run the feedback function on the given record. The app that
produced the record is also required to determine input/output argument
names.
| PARAMETER | DESCRIPTION |
|---|---|
app |
The app that produced the record. This can be AppDefinition or a jsonized AppDefinition. It will be jsonized if it is not already.
TYPE:
|
record |
The record to evaluate the feedback on. |
source_data |
Additional data to select from when extracting feedback function arguments. |
**kwargs |
Any additional keyword arguments are used to set or override selected feedback function inputs. |
| RETURNS | DESCRIPTION |
|---|---|
FeedbackResult
|
A FeedbackResult object with the result of the feedback function. |
extract_selection(app: Optional[Union[AppDefinition, JSON]] = None, record: Optional[Record] = None, source_data: Optional[Dict] = None) -> Iterable[Dict[str, Any]]
Given the app that produced the given record, extract from record
the values that will be sent as arguments to the implementation as
specified by self.selectors. Additional data to select from can be
provided in source_data. All args are optional. If a
Record is specified, its calls are
laid out as app (see
layout_calls_as_app).
Feedback-defining utilities¶
rag_triad(provider: LLMProvider, question: Optional[Lens] = None, answer: Optional[Lens] = None, context: Optional[Lens] = None) -> Dict[str, Feedback]
Create a triad of feedback functions for evaluating context retrieval generation steps.
If a particular lens is not provided, the relevant selectors will be missing. These can be filled in later or the triad can be used for rails feedback actions whick fill in the selectors based on specification from within colang.
| PARAMETER | DESCRIPTION |
|---|---|
provider |
The provider to use for implementing the feedback functions.
TYPE:
|
question |
Selector for the question part. |
answer |
Selector for the answer part. |
context |
Selector for the context part. |
Feedback-related types and containers¶
module-attribute
¶
Signature of feedback implementations.
Those take in any number of arguments and return either a single float or a float and a dictionary (of metadata).
module-attribute
¶
Signature of aggregation functions.
Serializable feedback-related classes.
Classes¶
Utilities for creating selectors using Lens and aliases/shortcuts.
Attributes¶
class-attribute
instance-attribute
¶
Selector for the tru wrapper (TruLlama, TruChain, etc.).
class-attribute
instance-attribute
¶
RecordInput: Query = main_input
Selector for the main app input.
class-attribute
instance-attribute
¶
RecordOutput: Query = main_output
Selector for the main app output.
class-attribute
instance-attribute
¶
RecordCalls: Query = app
Selector for the calls made by the wrapped app.
Layed out by path into components.
class-attribute
instance-attribute
¶
RecordCall: Query = calls[-1]
Selector for the first called method (last to return).
class-attribute
instance-attribute
¶
RecordArgs: Query = args
Selector for the whole set of inputs/arguments to the first called / last method call.
class-attribute
instance-attribute
¶
RecordRets: Query = rets
Selector for the whole output of the first called / last returned method call.
Functions¶
staticmethod
¶
If select names in method as the last attribute, extract the method name
and the selector without the final method name.
staticmethod
¶
If the given selector qualifies record or app, remove that qualification.
Mode of feedback evaluation.
Specify this using the feedback_mode to App constructors.
Attributes¶
class-attribute
instance-attribute
¶
NONE = 'none'
No evaluation will happen even if feedback functions are specified.
class-attribute
instance-attribute
¶
WITH_APP = 'with_app'
Try to run feedback functions immediately and before app returns a record.
class-attribute
instance-attribute
¶
WITH_APP_THREAD = 'with_app_thread'
Try to run feedback functions in the same process as the app but after it produces a record.
class-attribute
instance-attribute
¶
DEFERRED = 'deferred'
Evaluate later via the process started by
tru.start_deferred_feedback_evaluator.
Bases: Enum
For deferred feedback evaluation, these values indicate status of evaluation.
Attributes¶
class-attribute
instance-attribute
¶
RUNNING = 'running'
Once queued/started, status is updated to "running".
class-attribute
instance-attribute
¶
SKIPPED = 'skipped'
This feedback was skipped.
This can be because because it had an if_exists selector and did not
select anything or it has a selector that did not select anything the
on_missing was set to warn or ignore.
How to handle missing parameters in feedback function calls.
This is specifically for the case were a feedback function has a selector that selects something that does not exist in a record/app.
Bases: SerialModel
Invocations of feedback function results in one of these instances.
Note that a single Feedback instance might require more than one call.
Attributes¶
Bases: SerialModel
Feedback results for a single Feedback instance.
This might involve multiple feedback function calls. Typically you should not be constructing these objects yourself except for the cases where you'd like to log human feedback.
| ATTRIBUTE | DESCRIPTION |
|---|---|
feedback_result_id |
Unique identifier for this result.
TYPE:
|
record_id |
Record over which the feedback was evaluated.
TYPE:
|
feedback_definition_id |
The id of the FeedbackDefinition which was evaluated to get this result.
TYPE:
|
last_ts |
Last timestamp involved in the evaluation.
TYPE:
|
status |
For deferred feedback evaluation, the status of the evaluation.
TYPE:
|
cost |
Cost of the evaluation.
TYPE:
|
name |
Given name of the feedback.
TYPE:
|
calls |
Individual feedback function invocations.
TYPE:
|
result |
Final result, potentially aggregating multiple calls.
TYPE:
|
error |
Error information if there was an error.
TYPE:
|
multi_result |
TODO: doc
TYPE:
|
Attributes¶
class-attribute
instance-attribute
¶
status: FeedbackResultStatus = NONE
For deferred feedback evaluation, the status of the evaluation.
How to collect arguments for feedback function calls.
Note that this applies only to cases where selectors pick out more than one
thing for feedback function arguments. This option is used for the field
combinations of
FeedbackDefinition and can be
specified with
Feedback.aggregate.
Attributes¶
class-attribute
instance-attribute
¶
ZIP = 'zip'
Match argument values per position in produced values.
Example
If the selector for arg1 generates values 0, 1, 2 and one for arg2
generates values "a", "b", "c", the feedback function will be called 3
times with kwargs:
{'arg1': 0, arg2: "a"},{'arg1': 1, arg2: "b"},{'arg1': 2, arg2: "c"}
If the quantities of items in the various generators do not match, the result will have only as many combinations as the generator with the fewest items as per python zip (strict mode is not used).
Note that selectors can use
Lens collect() to name a single (list)
value instead of multiple values.
class-attribute
instance-attribute
¶
PRODUCT = 'product'
Evaluate feedback on all combinations of feedback function arguments.
Example
If the selector for arg1 generates values 0, 1 and the one for
arg2 generates values "a", "b", the feedback function will be called
4 times with kwargs:
{'arg1': 0, arg2: "a"},{'arg1': 0, arg2: "b"},{'arg1': 1, arg2: "a"},{'arg1': 1, arg2: "b"}
See itertools.product for more.
Note that selectors can use
Lens collect() to name a single (list)
value instead of multiple values.
Bases: WithClassInfo, SerialModel, Hashable
Serialized parts of a feedback function.
The non-serialized parts are in the Feedback class.
Attributes¶
class-attribute
instance-attribute
¶
Implementation serialization.
class-attribute
instance-attribute
¶
Aggregator method serialization.
class-attribute
instance-attribute
¶
combinations: Optional[FeedbackCombinations] = PRODUCT
Mode of combining selected values to produce arguments to each feedback function call.
class-attribute
instance-attribute
¶
Only execute the feedback function if the following selector names something that exists in a record/app.
Can use this to evaluate conditionally on presence of some calls, for example. Feedbacks skipped this way will have a status of FeedbackResultStatus.SKIPPED.
class-attribute
instance-attribute
¶
if_missing: FeedbackOnMissingParameters = ERROR
How to handle missing parameters in feedback function calls.
class-attribute
instance-attribute
¶
An optional name. Only will affect displayed tables.
class-attribute
instance-attribute
¶
Feedback result magnitude interpretation.
instance-attribute
¶
feedback_definition_id: FeedbackDefinitionID = feedback_definition_id
Id, if not given, uniquely determined from content.