💾 Record

trulens_eval.schema.record.Record

Bases: SerialModel, Hashable

The record of a single main method call.

Note

This class will be renamed to Trace in the future.

Attributes

app_id instance-attribute

app_id: AppID

The app that produced this record.

cost class-attribute instance-attribute

cost: Optional[Cost] = None

Costs associated with the record.

perf class-attribute instance-attribute

perf: Optional[Perf] = None

Performance information.

ts class-attribute instance-attribute

ts: datetime = Field(default_factory=now)

Timestamp of last update.

This is usually set whenever a record is changed in any way.

tags class-attribute instance-attribute

tags: Optional[str] = ''

Tags for the record.

meta class-attribute instance-attribute

meta: Optional[JSON] = None

Metadata for the record.

main_input class-attribute instance-attribute

main_input: Optional[JSON] = None

The app's main input.

main_output class-attribute instance-attribute

main_output: Optional[JSON] = None

The app's main output if there was no error.

main_error class-attribute instance-attribute

main_error: Optional[JSON] = None

The app's main error if there was an error.

calls class-attribute instance-attribute

calls: List[RecordAppCall] = []

The collection of calls recorded.

Note that these can be converted into a json structure with the same paths as the app that generated this record via layout_calls_as_app.

feedback_and_future_results class-attribute instance-attribute

feedback_and_future_results: Optional[List[Tuple[FeedbackDefinition, Future[FeedbackResult]]]] = Field(None, exclude=True)

Map of feedbacks to the futures for of their results.

These are only filled for records that were just produced. This will not be filled in when read from database. Also, will not fill in when using FeedbackMode.DEFERRED.

feedback_results class-attribute instance-attribute

feedback_results: Optional[List[Future[FeedbackResult]]] = Field(None, exclude=True)

Only the futures part of the above for backwards compatibility.

record_id instance-attribute

record_id: RecordID = record_id

Unique identifier for this record.

Functions

wait_for_feedback_results

wait_for_feedback_results() -> Dict[FeedbackDefinition, FeedbackResult]

Wait for feedback results to finish.

RETURNS	DESCRIPTION
Dict[FeedbackDefinition, FeedbackResult]	A mapping of feedback functions to their results.

layout_calls_as_app

layout_calls_as_app() -> Munch

Layout the calls in this record into the structure that follows that of the app that created this record.

This uses the paths stored in each RecordAppCall which are paths into the app.

Note: We cannot create a validated AppDefinition class (or subclass) object here as the layout of records differ in these ways:

• Records do not include anything that is not an instrumented method hence have most of the structure of a app missing.

• Records have RecordAppCall as their leafs where method definitions would be in the AppDefinition structure.

trulens_eval.schema.record.RecordAppCall

Bases: SerialModel

Info regarding each instrumented method call.

Attributes

stack instance-attribute

stack: List[RecordAppCallMethod]

Call stack but only containing paths of instrumented apps/other objects.

args instance-attribute

args: JSON

Arguments to the instrumented method.

rets class-attribute instance-attribute

rets: Optional[JSON] = None

Returns of the instrumented method if successful.

Sometimes this is a dict, sometimes a sequence, and sometimes a base value.

error class-attribute instance-attribute

error: Optional[str] = None

Error message if call raised exception.

perf class-attribute instance-attribute

perf: Optional[Perf] = None

Timestamps tracking entrance and exit of the instrumented method.

pid instance-attribute

pid: int

Process id.

tid instance-attribute

tid: int

Thread id.

Functions

top

top() -> RecordAppCallMethod

The top of the stack.

method

method() -> Method

The method at the top of the stack.

trulens_eval.schema.record.RecordAppCallMethod

Bases: SerialModel

Method information for the stacks inside RecordAppCall.

Attributes

path instance-attribute

path: Lens

Path to the method in the app's structure.

method instance-attribute

method: Method

The method that was called.

trulens_eval.schema.base.Cost

Bases: SerialModel, BaseModel

Costs associated with some call or set of calls.

Attributes

n_requests class-attribute instance-attribute

n_requests: int = 0

Number of requests.

n_successful_requests class-attribute instance-attribute

n_successful_requests: int = 0

Number of successful requests.

n_classes class-attribute instance-attribute

n_classes: int = 0

Number of class scores retrieved.

n_tokens class-attribute instance-attribute

n_tokens: int = 0

Total tokens processed.

n_stream_chunks class-attribute instance-attribute

n_stream_chunks: int = 0

In streaming mode, number of chunks produced.

n_prompt_tokens class-attribute instance-attribute

n_prompt_tokens: int = 0

Number of prompt tokens supplied.

n_completion_tokens class-attribute instance-attribute

n_completion_tokens: int = 0

Number of completion tokens generated.

cost class-attribute instance-attribute

cost: float = 0.0

Cost in USD.

trulens_eval.schema.base.Perf

Bases: SerialModel, BaseModel

Performance information.

Presently only the start and end times, and thus latency.

Attributes

start_time instance-attribute

start_time: datetime

Datetime before the recorded call.

end_time instance-attribute

end_time: datetime

Datetime after the recorded call.

latency property

latency

Latency in seconds.

Functions

min staticmethod

min()

Zero-length span with start and end times at the minimum datetime.

now staticmethod

now(latency: Optional[timedelta] = None) -> Perf

Create a Perf instance starting now and ending now plus latency.

PARAMETER	DESCRIPTION
latency	Latency in seconds. If given, end time will be now plus latency. Otherwise end time will be a minimal interval plus start_time. TYPE: Optional[timedelta] DEFAULT: None