Skip to content

๐„ข Instruments

trulens_eval.instruments

Instrumentation

This module contains the core of the app instrumentation scheme employed by trulens_eval to track and record apps. These details should not be relevant for typical use cases.

Classes

WithInstrumentCallbacks

Abstract definition of callbacks invoked by Instrument during instrumentation or when instrumented methods are called.

Needs to be mixed into App.

Functions
on_method_instrumented
on_method_instrumented(obj: object, func: Callable, path: Lens)

Callback to be called by instrumentation system for every function requested to be instrumented.

Given are the object of the class in which func belongs (i.e. the "self" for that function), the func itsels, and the path of the owner object in the app hierarchy.

PARAMETER DESCRIPTION
obj

The object of the class in which func belongs (i.e. the "self" for that method).

TYPE: object

func

The function that was instrumented. Expects the unbound version (self not yet bound).

TYPE: Callable

path

The path of the owner object in the app hierarchy.

TYPE: Lens

get_method_path
get_method_path(obj: object, func: Callable) -> Lens

Get the path of the instrumented function func, a member of the class of obj relative to this app.

PARAMETER DESCRIPTION
obj

The object of the class in which func belongs (i.e. the "self" for that method).

TYPE: object

func

The function that was instrumented. Expects the unbound version (self not yet bound).

TYPE: Callable

get_methods_for_func
get_methods_for_func(func: Callable) -> Iterable[Tuple[int, Callable, Lens]]

Get the methods (rather the inner functions) matching the given func and the path of each.

PARAMETER DESCRIPTION
func

The function to match.

TYPE: Callable

on_new_record
on_new_record(func: Callable)

Called by instrumented methods in cases where they cannot find a record call list in the stack. If we are inside a context manager, return a new call list.

on_add_record
on_add_record(ctx: 'RecordingContext', func: Callable, sig: Signature, bindings: BoundArguments, ret: Any, error: Any, perf: Perf, cost: Cost, existing_record: Optional[Record] = None)

Called by instrumented methods if they are root calls (first instrumned methods in a call stack).

PARAMETER DESCRIPTION
ctx

The context of the recording.

TYPE: 'RecordingContext'

func

The function that was called.

TYPE: Callable

sig

The signature of the function.

TYPE: Signature

bindings

The bound arguments of the function.

TYPE: BoundArguments

ret

The return value of the function.

TYPE: Any

error

The error raised by the function if any.

TYPE: Any

perf

The performance of the function.

TYPE: Perf

cost

The cost of the function.

TYPE: Cost

existing_record

If the record has already been produced (i.e. because it was an awaitable), it can be passed here to avoid re-creating it.

TYPE: Optional[Record] DEFAULT: None

Instrument

Bases: object

Instrumentation tools.

Attributes
INSTRUMENT class-attribute instance-attribute
INSTRUMENT = '__tru_instrumented'

Attribute name to be used to flag instrumented objects/methods/others.

APPS class-attribute instance-attribute
APPS = '__tru_apps'

Attribute name for storing apps that expect to be notified of calls.

Classes
Default

Default instrumentation configuration.

Additional components are included in subclasses of Instrument.

Attributes
MODULES class-attribute instance-attribute
MODULES = {'trulens_eval.'}

Modules (by full name prefix) to instrument.

CLASSES class-attribute instance-attribute
CLASSES = set([Feedback])

Classes to instrument.

METHODS class-attribute instance-attribute
METHODS: Dict[str, ClassFilter] = {'__call__': Feedback}

Methods to instrument.

Methods matching name have to pass the filter to be instrumented.

Functions
print_instrumentation
print_instrumentation() -> None

Print out description of the modules, classes, methods this class will instrument.

to_instrument_object
to_instrument_object(obj: object) -> bool

Determine whether the given object should be instrumented.

to_instrument_class
to_instrument_class(cls: type) -> bool

Determine whether the given class should be instrumented.

to_instrument_module
to_instrument_module(module_name: str) -> bool

Determine whether a module with the given (full) name should be instrumented.

tracked_method_wrapper
tracked_method_wrapper(query: Lens, func: Callable, method_name: str, cls: type, obj: object)

Wrap a method to capture its inputs/outputs/errors.

instrument_method
instrument_method(method_name: str, obj: Any, query: Lens)

Instrument a method.

instrument_class
instrument_class(cls)

Instrument the given class cls's new method.

This is done so we can be aware when new instances are created and is needed for wrapped methods that dynamically create instances of classes we wish to instrument. As they will not be visible at the time we wrap the app, we need to pay attention to new to make a note of them when they are created and the creator's path. This path will be used to place these new instances in the app json structure.

instrument_object
instrument_object(obj, query: Lens, done: Optional[Set[int]] = None)

Instrument the given object obj and its components.

instrument_bound_methods
instrument_bound_methods(obj: object, query: Lens)

Instrument functions that may be bound methods.

Some apps include either anonymous functions or manipulates methods that have self bound already. Our other instrumentation cannot handle those cases.

Warning

Experimental work in progress.

AddInstruments

Utilities for adding more things to default instrumentation filters.

Functions
method classmethod
method(of_cls: type, name: str) -> None

Add the class with a method named name, its module, and the method name to the Default instrumentation walk filters.

methods classmethod
methods(of_cls: type, names: Iterable[str]) -> None

Add the class with methods named names, its module, and the named methods to the Default instrumentation walk filters.

instrument

Bases: AddInstruments

Decorator for marking methods to be instrumented in custom classes that are wrapped by App.

Functions

class_filter_disjunction

class_filter_disjunction(f1: ClassFilter, f2: ClassFilter) -> ClassFilter

Create a disjunction of two class filters.

PARAMETER DESCRIPTION
f1

The first filter.

TYPE: ClassFilter

f2

The second filter.

TYPE: ClassFilter

class_filter_matches

class_filter_matches(f: ClassFilter, obj: Union[Type, object]) -> bool

Check whether given object matches a class-based filter.

A class-based filter here means either a type to match against object (isinstance if object is not a type or issubclass if object is a type), or a tuple of types to match against interpreted disjunctively.

PARAMETER DESCRIPTION
f

The filter to match against.

TYPE: ClassFilter

obj

The object to match against. If type, uses issubclass to match. If object, uses isinstance to match against filters of Type or Tuple[Type].

TYPE: Union[Type, object]