𝄢 Instruments

trulens_eval.instruments

Instrumentation

This module contains the core of the app instrumentation scheme employed by trulens_eval to track and record apps. These details should not be relevant for typical use cases.

Classes

WithInstrumentCallbacks

Abstract definition of callbacks invoked by Instrument during instrumentation or when instrumented methods are called.

Needs to be mixed into App.

Functions

on_method_instrumented

on_method_instrumented(obj: object, func: Callable, path: Lens)

Callback to be called by instrumentation system for every function requested to be instrumented.

Given are the object of the class in which func belongs (i.e. the "self" for that function), the func itsels, and the path of the owner object in the app hierarchy.

PARAMETER	DESCRIPTION
obj	The object of the class in which func belongs (i.e. the "self" for that method). TYPE: object
func	The function that was instrumented. Expects the unbound version (self not yet bound). TYPE: Callable
path	The path of the owner object in the app hierarchy. TYPE: Lens

get_method_path

get_method_path(obj: object, func: Callable) -> Lens

Get the path of the instrumented function func, a member of the class of obj relative to this app.

PARAMETER	DESCRIPTION
obj	The object of the class in which func belongs (i.e. the "self" for that method). TYPE: object
func	The function that was instrumented. Expects the unbound version (self not yet bound). TYPE: Callable

get_methods_for_func

get_methods_for_func(func: Callable) -> Iterable[Tuple[int, Callable, Lens]]

Get the methods (rather the inner functions) matching the given func and the path of each.

PARAMETER	DESCRIPTION
func	The function to match. TYPE: Callable

on_new_record

on_new_record(func: Callable)

Called by instrumented methods in cases where they cannot find a record call list in the stack. If we are inside a context manager, return a new call list.

on_add_record

on_add_record(ctx: 'RecordingContext', func: Callable, sig: Signature, bindings: BoundArguments, ret: Any, error: Any, perf: Perf, cost: Cost, existing_record: Optional[Record] = None)

Called by instrumented methods if they are root calls (first instrumned methods in a call stack).

PARAMETER	DESCRIPTION
ctx	The context of the recording. TYPE: 'RecordingContext'
func	The function that was called. TYPE: Callable
sig	The signature of the function. TYPE: Signature
bindings	The bound arguments of the function. TYPE: BoundArguments
ret	The return value of the function. TYPE: Any
error	The error raised by the function if any. TYPE: Any
perf	The performance of the function. TYPE: Perf
cost	The cost of the function. TYPE: Cost
existing_record	If the record has already been produced (i.e. because it was an awaitable), it can be passed here to avoid re-creating it. TYPE: Optional[Record] DEFAULT: None

Instrument

Bases: object

Instrumentation tools.

Attributes

INSTRUMENT class-attribute instance-attribute

INSTRUMENT = '__tru_instrumented'

Attribute name to be used to flag instrumented objects/methods/others.

APPS class-attribute instance-attribute

APPS = '__tru_apps'

Attribute name for storing apps that expect to be notified of calls.

Classes

Default

Default instrumentation configuration.

Additional components are included in subclasses of Instrument.

Attributes

MODULES class-attribute instance-attribute

MODULES = {'trulens_eval.'}

Modules (by full name prefix) to instrument.

CLASSES class-attribute instance-attribute

CLASSES = set([Feedback])

Classes to instrument.

METHODS class-attribute instance-attribute

METHODS: Dict[str, ClassFilter] = {'__call__': Feedback}

Methods to instrument.

Methods matching name have to pass the filter to be instrumented.

Functions

print_instrumentation

print_instrumentation() -> None

Print out description of the modules, classes, methods this class will instrument.

to_instrument_object

to_instrument_object(obj: object) -> bool

Determine whether the given object should be instrumented.

to_instrument_class

to_instrument_class(cls: type) -> bool

Determine whether the given class should be instrumented.

to_instrument_module

to_instrument_module(module_name: str) -> bool

Determine whether a module with the given (full) name should be instrumented.

tracked_method_wrapper

tracked_method_wrapper(query: Lens, func: Callable, method_name: str, cls: type, obj: object)

Wrap a method to capture its inputs/outputs/errors.

instrument_method

instrument_method(method_name: str, obj: Any, query: Lens)

Instrument a method.

instrument_class

instrument_class(cls)

Instrument the given class cls's new method.

This is done so we can be aware when new instances are created and is needed for wrapped methods that dynamically create instances of classes we wish to instrument. As they will not be visible at the time we wrap the app, we need to pay attention to new to make a note of them when they are created and the creator's path. This path will be used to place these new instances in the app json structure.

instrument_object

instrument_object(obj, query: Lens, done: Optional[Set[int]] = None)

Instrument the given object obj and its components.

instrument_bound_methods

instrument_bound_methods(obj: object, query: Lens)

Instrument functions that may be bound methods.

Some apps include either anonymous functions or manipulates methods that have self bound already. Our other instrumentation cannot handle those cases.

Warning

Experimental work in progress.

AddInstruments

Utilities for adding more things to default instrumentation filters.

Functions

method classmethod

method(of_cls: type, name: str) -> None

Add the class with a method named name, its module, and the method name to the Default instrumentation walk filters.

methods classmethod

methods(of_cls: type, names: Iterable[str]) -> None

Add the class with methods named names, its module, and the named methods to the Default instrumentation walk filters.

instrument

Bases: AddInstruments

Decorator for marking methods to be instrumented in custom classes that are wrapped by App.

Functions

class_filter_disjunction

class_filter_disjunction(f1: ClassFilter, f2: ClassFilter) -> ClassFilter

Create a disjunction of two class filters.

PARAMETER	DESCRIPTION
f1	The first filter. TYPE: ClassFilter
f2	The second filter. TYPE: ClassFilter

class_filter_matches

class_filter_matches(f: ClassFilter, obj: Union[Type, object]) -> bool

Check whether given object matches a class-based filter.

A class-based filter here means either a type to match against object (isinstance if object is not a type or issubclass if object is a type), or a tuple of types to match against interpreted disjunctively.

PARAMETER	DESCRIPTION
f	The filter to match against. TYPE: ClassFilter
obj	The object to match against. If type, uses issubclass to match. If object, uses isinstance to match against filters of Type or Tuple[Type]. TYPE: Union[Type, object]