📓 Instrumentation Overview¶
TruLens is a framework that helps you instrument and evaluate LLM apps including RAGs and agents.
Because TruLens is tech-agnostic, we offer a few different tools for instrumentation.
- TruCustomApp gives you the most power to instrument a custom LLM app, and
provides the
instrument
method. - TruBasicApp is a simple interface to capture the input and output of a basic LLM app.
- TruChain instruments LangChain apps. Read more.
- TruLlama instruments LlamaIndex apps. Read more.
- TruRails instruments NVIDIA Nemo Guardrails apps. Read more.
In any framework you can track (and evaluate) the intputs, outputs and instrumented internals, along with a wide variety of usage metrics and metadata, detailed below:
Usage Metrics¶
- Number of requests (n_requests)
- Number of successful ones (n_successful_requests)
- Number of class scores retrieved (n_classes)
- Total tokens processed (n_tokens)
- In streaming mode, number of chunks produced (n_stream_chunks)
- Number of prompt tokens supplied (n_prompt_tokens)
- Number of completion tokens generated (n_completion_tokens)
- Cost in USD (cost)
Read more about Usage Tracking in [Cost API Reference][trulens_eval.schema.base.Cost].
App Metadata¶
- App ID (app_id) - user supplied string or automatically generated hash
- Tags (tags) - user supplied string
- Model metadata - user supplied json
Record Metadata¶
- Record ID (record_id) - automatically generated, track individual application calls
- Timestamp (ts) - automatcially tracked, the timestamp of the application call
- Latency (latency) - the difference between the application call start and end time.
Instrumenting LLM applications¶
Evaluating LLM applications often requires access to the internals of an app,
such as retrieved context. To gain access to these internals, TruLens provides
the instrument
method. In cases where you have access to the classes and
methods required, you can add the @instrument
decorator to any method you wish
to instrument. See a usage example below:
Using the @instrument
decorator¶
from trulens_eval.tru_custom_app import instrument
class RAG_from_scratch:
@instrument
def retrieve(self, query: str) -> list:
"""
Retrieve relevant text from vector store.
"""
@instrument
def generate_completion(self, query: str, context_str: list) -> str:
"""
Generate answer from context.
"""
@instrument
def query(self, query: str) -> str:
"""
Retrieve relevant text given a query, and then generate an answer from the context.
"""
In cases you do not have access to a class to make the necessary decorations for
tracking, you can instead use one of the static methods of instrument, for
example, the alterative for making sure the custom retriever gets instrumented
is via instrument.method
. See a usage example below:
Using the instrument.method
¶
from trulens_eval.tru_custom_app import instrument
from somepackage.from custom_retriever import CustomRetriever
instrument.method(CustomRetriever, "retrieve_chunks")
# ... rest of the custom class follows ...
Read more about instrumenting custom class applications in the API Reference
Tracking input-output applications¶
For basic tracking of inputs and outputs, TruBasicApp
can be used for instrumentation.
Suppose you have a generic text-to-text application as follows:
def custom_application(prompt: str) -> str:
return "a response"
After creating the application, TruBasicApp allows you to instrument it in one line of code:
from trulens_eval import TruBasicApp
basic_app_recorder = TruBasicApp(custom_application, app_id="Custom Application v1")
Then, you can operate the application like normal:
with basic_app_recorder as recording:
basic_app_recorder.app("What is the phone number for HR?")
Read more about TruBasicApp in the API reference or check out the text2text quickstart.
If instead, you're looking to use TruLens with a more complex custom application, you can use TruCustom.
For more information, plese read more about TruCustom in the API Reference
For frameworks with deep integrations, TruLens can expose additional internals of the application for tracking. See TruChain and TruLlama for more details.