Tru Virtual¶
Bases: Record
Virtual records for virtual apps.
Many arguments are filled in by default values if not provided. See Record for all arguments. Listing here is only for those which are required for this method or filled with default values.
| PARAMETER | DESCRIPTION |
|---|---|
calls |
A dictionary of calls to be recorded. The keys are selectors and the values are dictionaries with the keys listed in the next section. |
cost |
Defaults to zero cost. |
perf |
Defaults to time spanning the processing of this virtual record. Note that individual calls also include perf. Time span is extended to make sure it is not of duration zero. |
Call values are dictionaries containing arguments to RecordAppCall constructor. Values can also be lists of the same. This happens in non-virtual apps when the same method is recorded making multiple calls in a single app invocation. The following defaults are used if not provided.
| PARAMETER | TYPE | DEFAULT |
|---|---|---|
stack |
List[RecordAppCallMethod] | Two frames: a root call followed by a call by virtual_object, method name derived from the last element of the selector of this call. |
args |
JSON | [] |
rets |
JSON | [] |
perf |
Perf | Time spanning the processing of this virtual call. |
pid |
int | 0 |
tid |
int | 0 |
Bases: dict
A dictionary meant to represent the components of a virtual app.
TruVirtual will refer to this class as the wrapped app. All calls will be
under VirtualApp.root
Bases: App
Recorder for virtual apps.
Virtual apps are data only in that they cannot be executed but for whom previously-computed results can be added using add_record. The VirtualRecord class may be useful for creating records for this. Fields used by non-virtual apps can be specified here, notably:
See App and AppDefinition for constructor arguments.
The app field.¶
You can store any information you would like by passing in a dictionary to
TruVirtual in the app field. This may involve an index of components or
versions, or anything else. You can refer to these values for evaluating
feedback.
Usage
You can use VirtualApp to create the app structure or a plain
dictionary. Using VirtualApp lets you use Selectors to define components:
virtual_app = VirtualApp()
virtual_app[Select.RecordCalls.llm.maxtokens] = 1024
Example
virtual_app = dict(
llm=dict(
modelname="some llm component model name"
),
template="information about the template I used in my app",
debug="all of these fields are completely optional"
)
virtual = TruVirtual(
app_id="my_virtual_app",
app=virtual_app
)
Attributes¶
class-attribute
instance-attribute
¶
feedback_definitions: Sequence[FeedbackDefinition] = []
Feedback functions to evaluate on each record.
class-attribute
instance-attribute
¶
feedback_mode: FeedbackMode = WITH_APP_THREAD
How to evaluate feedback functions upon producing a record.
class-attribute
instance-attribute
¶
initial_app_loader_dump: Optional[SerialBytes] = None
Serialization of a function that loads an app.
Dump is of the initial app state before any invocations. This can be used to create a new session.
Warning
Experimental work in progress.
instance-attribute
¶
app_extra_json: JSON
Info to store about the app and to display in dashboard.
This can be used even if app itself cannot be serialized. app_extra_json,
then, can stand in place for whatever data the user might want to keep track
of about the app.
class-attribute
instance-attribute
¶
Feedback functions to evaluate on each record.
class-attribute
instance-attribute
¶
Workspace manager.
If this is not povided, a singleton Tru will be made (if not already) and used.
class-attribute
instance-attribute
¶
Database interface.
If this is not provided, a singleton SQLAlchemyDB will be made (if not already) and used.
class-attribute
instance-attribute
¶
recording_contexts: ContextVar[RecordingContext] = Field(None, exclude=True)
Sequnces of records produced by the this class used as a context manager are stored in a RecordingContext.
Using a context var so that context managers can be nested.
class-attribute
instance-attribute
¶
Mapping of instrumented methods (by id(.) of owner object and the function) to their path in this app.
class-attribute
instance-attribute
¶
records_with_pending_feedback_results: Queue[Record] = Field(exclude=True, default_factory=lambda: Queue(maxsize=1024))
Records produced by this app which might have yet to finish feedback runs.
class-attribute
instance-attribute
¶
Thread for manager of pending feedback results queue.
See _manage_pending_feedback_results.
class-attribute
instance-attribute
¶
selector_check_warning: bool = False
Selector checking is disabled for virtual apps.
class-attribute
instance-attribute
¶
selector_nocheck: bool = True
The selector check must be disabled for virtual apps.
This is because methods that could be called are not known in advance of creating virtual records.
Functions¶
Called by instrumentation system for every function requested to be instrumented by this app.
Get the path of the instrumented function method relative to this app.
Get the methods (rather the inner functions) matching the given func
and the path of each.
on_new_record(func) -> Iterable[RecordingContext]
Called at the start of record creation.
on_add_record(ctx: RecordingContext, func: Callable, sig: Signature, bindings: BoundArguments, ret: Any, error: Any, perf: Perf, cost: Cost, existing_record: Optional[Record] = None) -> Record
Called by instrumented methods if they use _new_record to construct a record call list.
staticmethod
¶
load(obj, *args, **kwargs)
Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.
classmethod
¶
model_validate(*args, **kwargs) -> Any
Deserialized a jsonized version of the app into the instance of the class it was serialized from.
Note
This process uses extra information stored in the jsonized object and handled by WithClassInfo.
staticmethod
¶
continue_session(app_definition_json: JSON, app: Any) -> AppDefinition
Instantiate the given app with the given state
app_definition_json.
Warning
This is an experimental feature with ongoing work.
| PARAMETER | DESCRIPTION |
|---|---|
app_definition_json |
The json serialized app.
TYPE:
|
app |
The app to continue the session with.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
AppDefinition
|
A new |
staticmethod
¶
new_session(app_definition_json: JSON, initial_app_loader: Optional[Callable] = None) -> AppDefinition
Create an app instance at the start of a session.
Warning
This is an experimental feature with ongoing work.
Create a copy of the json serialized app with the enclosed app being initialized to its initial state before any records are produced (i.e. blank memory).
staticmethod
¶
get_loadable_apps()
Gets a list of all of the loadable apps.
Warning
This is an experimental feature with ongoing work.
This is those that have initial_app_loader_dump set.
wait_for_feedback_results() -> None
Wait for all feedbacks functions to complete.
This applies to all feedbacks on all records produced by this app. This call will block until finished and if new records are produced while this is running, it will include them.
classmethod
¶
Try to find retriever components in the given app and return a lens to
access the retrieved contexts that would appear in a record were these
components to execute.
If available, a single text to a single text invocation of this app.
async
¶
If available, a single text to a single text invocation of this app.
main_input(func: Callable, sig: Signature, bindings: BoundArguments) -> JSON
Determine the main input string for the given function func with
signature sig if it is to be called with the given bindings
bindings.
main_output(func: Callable, sig: Signature, bindings: BoundArguments, ret: Any) -> JSON
Determine the main out string for the given function func with
signature sig after it is called with the given bindings and has
returned ret.
async
¶
awith_(func: CallableMaybeAwaitable[A, T], *args, **kwargs) -> T
Call the given async func with the given *args and **kwargs while
recording, producing func results. The record of the computation is
available through other means like the database or dashboard. If you
need a record of this execution immediately, you can use awith_record
or the App as a context mananger instead.
async
¶
with_(func: Callable[[A], T], *args, **kwargs) -> T
Call the given async func with the given *args and **kwargs while
recording, producing func results. The record of the computation is
available through other means like the database or dashboard. If you
need a record of this execution immediately, you can use awith_record
or the App as a context mananger instead.
with_record(func: Callable[[A], T], *args, record_metadata: JSON = None, **kwargs) -> Tuple[T, Record]
Call the given func with the given *args and **kwargs, producing
its results as well as a record of the execution.
async
¶
awith_record(func: Callable[[A], Awaitable[T]], *args, record_metadata: JSON = None, **kwargs) -> Tuple[T, Record]
Call the given func with the given *args and **kwargs, producing
its results as well as a record of the execution.
dummy_record(cost: Cost = mod_base_schema.Cost(), perf: Perf = mod_base_schema.Perf.now(), ts: datetime = datetime.datetime.now(), main_input: str = 'main_input are strings.', main_output: str = 'main_output are strings.', main_error: str = 'main_error are strings.', meta: Dict = {'metakey': 'meta are dicts'}, tags: str = 'tags are strings') -> Record
Create a dummy record with some of the expected structure without actually invoking the app.
The record is a guess of what an actual record might look like but will be missing information that can only be determined after a call is made.
All args are Record fields except these:
- `record_id` is generated using the default id naming schema.
- `app_id` is taken from this recorder.
- `calls` field is constructed based on instrumented methods.
Iteration over instrumented components and their categories.
format_instrumented_methods() -> str
Build a string containing a listing of instrumented methods.
print_instrumented_components() -> None
Print instrumented components and their categories.
__init__(app: Optional[Union[VirtualApp, JSON]] = None, **kwargs: dict)
Virtual app for logging existing app results.
add_record(record: Record, feedback_mode: Optional[FeedbackMode] = None) -> Record
Add the given record to the database and evaluate any pre-specified feedbacks on it.
The class VirtualRecord may be useful for creating
records for virtual models. If feedback_mode is specified, will use
that mode for this record only.
module-attribute
¶
virtual_module = Module(package_name='trulens_eval', module_name='trulens_eval.tru_virtual')
Module to represent the module of virtual apps.
Virtual apps will record this as their module.
module-attribute
¶
virtual_class = Class(module=virtual_module, name='VirtualApp')
Class to represent the class of virtual apps.
Virtual apps will record this as their class.
module-attribute
¶
virtual_object = Obj(cls=virtual_class, id=0)
Object to represent instances of virtual apps.
Virtual apps will record this as their instance.
module-attribute
¶
virtual_method_root = Method(cls=virtual_class, obj=virtual_object, name='root')
Method call to represent the root call of virtual apps.
Virtual apps will record this as their root call.
module-attribute
¶
virtual_method_call = Method(cls=virtual_class, obj=virtual_object, name='method_name_not_set')
Method call to represent virtual app calls that do not provide this information.
Method name will be replaced by the last attribute in the selector provided by user.