Overview¶

TruLens provides a number of different instrumentation frameworks to allow you to inspect and evaluate the internals of your application and its associated records. In any framework you can track a wide variety of usage metrics and metadata, detailed below, along with the inputs and outputs of the application. For frameworks with deep integrations, TruLens can expose additional internals of the application for tracking.

What can you track?¶

Usage Metrics¶

• Number of requests (n_requests)
• Number of successful ones (n_successful_requests)
• Number of class scores retrieved (n_classes)
• Total tokens processed (n_tokens)
• In streaming mode, number of chunks produced (n_stream_chunks)
• Number of prompt tokens supplied (n_prompt_tokens)
• Number of completion tokens generated (n_completion_tokens)
• Cost in USD (cost)

App Metadata¶

• App ID (app_id) - user supplied string or automatically generated hash
• Tags (tags) - user supplied string
• Model metadata - user supplied json

Record Metadata¶

• Record ID (record_id) - automatically generated, track individual application calls
• Timestamp (ts) - automatcially tracked, the timestamp of the application call
• Latency (latency) - the difference between the application call start and end time.

Tracking custom applications¶

Outside of integrations, TruLens supports the instrumentation of any text-to-text application, including custom ones.

The way to track this type of application is through TruBasicApp.

Suppose you have a generic text-to-text application as follows:

In [ ]:

def custom_application(prompt: str) -> str:
    return "a response"

def custom_application(prompt: str) -> str: return "a response"

After creating the application, TruBasicApp allows you to instrument it in one line of code:

In [ ]:

from trulens_eval import TruBasicApp
basic_app_recorder = TruBasicApp(custom_application, app_id="Custom Application v1")

from trulens_eval import TruBasicApp basic_app_recorder = TruBasicApp(custom_application, app_id="Custom Application v1")

Then, you can operate the application like normal:

In [ ]:

with basic_app_recorder as recording:
    basic_app_recorder.app("What is the phone number for HR?")

with basic_app_recorder as recording: basic_app_recorder.app("What is the phone number for HR?")

Read more about TruBasicApp in the API reference or check out the non-framework quickstart.