📓 NeMo Guardrails Integration¶
TruLens provides TruRails, an integration with NeMo Guardrails apps to allow you to inspect and evaluate the internals of your application built using NeMo Guardrails. This is done through the instrumentation of key NeMo Guardrails classes. To see a list of classes instrumented, see Appendix: Instrumented Nemo Classes and Methods.
In addition to the default instrumentation, TruRails exposes the select_context method for evaluations that require access to retrieved context. Exposing select_context bypasses the need to know the json structure of your app ahead of time, and makes your evaluations re-usable across different apps.
Example Usage¶
Below is a quick example of usage. First, we'll create a standard Nemo app.
%%writefile config.yaml
# Adapted from NeMo-Guardrails/nemoguardrails/examples/bots/abc/config.yml
instructions:
- type: general
content: |
Below is a conversation between a user and a bot called the trulens Bot.
The bot is designed to answer questions about the trulens_eval python library.
The bot is knowledgeable about python.
If the bot does not know the answer to a question, it truthfully says it does not know.
sample_conversation: |
user "Hi there. Can you help me with some questions I have about trulens?"
express greeting and ask for assistance
bot express greeting and confirm and offer assistance
"Hi there! I'm here to help answer any questions you may have about the trulens. What would you like to know?"
models:
- type: main
engine: openai
model: gpt-3.5-turbo-instruct
Writing config.yaml
%%writefile config.co
# Adapted from NeMo-Guardrails/tests/test_configs/with_kb_openai_embeddings/config.co
define user ask capabilities
"What can you do?"
"What can you help me with?"
"tell me what you can do"
"tell me about you"
define bot inform capabilities
"I am an AI bot that helps answer questions about trulens_eval."
define flow
user ask capabilities
bot inform capabilities
Writing config.co
# Create a small knowledge base from the root README file.
! mkdir -p kb
! cp ../../../../README.md kb
from nemoguardrails import LLMRails, RailsConfig
from pprint import pprint
config = RailsConfig.from_path(".")
rails = LLMRails(config)
Fetching 7 files: 0%| | 0/7 [00:00<?, ?it/s]
To instrument an LLM chain, all that's required is to wrap it using TruChain.
from trulens_eval import TruRails
# instrument with TruRails
tru_recorder = TruRails(
rails,
app_id = "my first trurails app", # optional
)
To properly evaluate LLM apps we often need to point our evaluation at an internal step of our application, such as the retreived context. Doing so allows us to evaluate for metrics including context relevance and groundedness.
For Nemo applications with a knowledge base, select_context
can
be used to access the retrieved text for evaluation.
Example usage:
context = TruRails.select_context(rails)
f_context_relevance = (
Feedback(provider.qs_relevance)
.on_input()
.on(context)
.aggregate(np.mean)
)
For added flexibility, the select_context method is also made available through
trulens_eval.app.App
. This allows you to switch between frameworks without
changing your context selector:
from trulens_eval.app import App
context = App.select_context(rag_chain)
Appendix: Instrumented Nemo Classes and Methods¶
The modules, classes, and methods that trulens instruments can be retrieved from the appropriate Instrument subclass.
from trulens_eval.tru_rails import RailsInstrument
RailsInstrument().print_instrumentation()
Module langchain* Class langchain.agents.agent.BaseMultiActionAgent Method plan: (self, intermediate_steps: 'List[Tuple[AgentAction, str]]', callbacks: 'Callbacks' = None, **kwargs: 'Any') -> 'Union[List[AgentAction], AgentFinish]' Method aplan: (self, intermediate_steps: 'List[Tuple[AgentAction, str]]', callbacks: 'Callbacks' = None, **kwargs: 'Any') -> 'Union[List[AgentAction], AgentFinish]' Class langchain.agents.agent.BaseSingleActionAgent Method plan: (self, intermediate_steps: 'List[Tuple[AgentAction, str]]', callbacks: 'Callbacks' = None, **kwargs: 'Any') -> 'Union[AgentAction, AgentFinish]' Method aplan: (self, intermediate_steps: 'List[Tuple[AgentAction, str]]', callbacks: 'Callbacks' = None, **kwargs: 'Any') -> 'Union[AgentAction, AgentFinish]' Class langchain.chains.base.Chain Method __call__: (self, inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Union[List[langchain_core.callbacks.base.BaseCallbackHandler], langchain_core.callbacks.base.BaseCallbackManager, NoneType] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, run_name: Optional[str] = None, include_run_info: bool = False) -> Dict[str, Any] Method invoke: (self, input: Dict[str, Any], config: Optional[langchain_core.runnables.config.RunnableConfig] = None, **kwargs: Any) -> Dict[str, Any] Method ainvoke: (self, input: Dict[str, Any], config: Optional[langchain_core.runnables.config.RunnableConfig] = None, **kwargs: Any) -> Dict[str, Any] Method run: (self, *args: Any, callbacks: Union[List[langchain_core.callbacks.base.BaseCallbackHandler], langchain_core.callbacks.base.BaseCallbackManager, NoneType] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) -> Any Method arun: (self, *args: Any, callbacks: Union[List[langchain_core.callbacks.base.BaseCallbackHandler], langchain_core.callbacks.base.BaseCallbackManager, NoneType] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) -> Any Method _call: (self, inputs: Dict[str, Any], run_manager: Optional[langchain_core.callbacks.manager.CallbackManagerForChainRun] = None) -> Dict[str, Any] Method _acall: (self, inputs: Dict[str, Any], run_manager: Optional[langchain_core.callbacks.manager.AsyncCallbackManagerForChainRun] = None) -> Dict[str, Any] Method acall: (self, inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Union[List[langchain_core.callbacks.base.BaseCallbackHandler], langchain_core.callbacks.base.BaseCallbackManager, NoneType] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, run_name: Optional[str] = None, include_run_info: bool = False) -> Dict[str, Any] Class langchain.memory.chat_memory.BaseChatMemory Method save_context: (self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None Method clear: (self) -> None Class langchain_core.chat_history.BaseChatMessageHistory Class langchain_core.documents.base.Document Class langchain_core.language_models.base.BaseLanguageModel Class langchain_core.language_models.llms.BaseLLM Class langchain_core.load.serializable.Serializable Class langchain_core.memory.BaseMemory Method save_context: (self, inputs: 'Dict[str, Any]', outputs: 'Dict[str, str]') -> 'None' Method clear: (self) -> 'None' Class langchain_core.prompts.base.BasePromptTemplate Class langchain_core.retrievers.BaseRetriever Method _get_relevant_documents: (self, query: 'str', *, run_manager: 'CallbackManagerForRetrieverRun') -> 'List[Document]' Method get_relevant_documents: (self, query: 'str', *, callbacks: 'Callbacks' = None, tags: 'Optional[List[str]]' = None, metadata: 'Optional[Dict[str, Any]]' = None, run_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[Document]' Method aget_relevant_documents: (self, query: 'str', *, callbacks: 'Callbacks' = None, tags: 'Optional[List[str]]' = None, metadata: 'Optional[Dict[str, Any]]' = None, run_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[Document]' Method _aget_relevant_documents: (self, query: 'str', *, run_manager: 'AsyncCallbackManagerForRetrieverRun') -> 'List[Document]' Class langchain_core.runnables.base.RunnableSerializable Class langchain_core.tools.BaseTool Method _arun: (self, *args: 'Any', **kwargs: 'Any') -> 'Any' Method _run: (self, *args: 'Any', **kwargs: 'Any') -> 'Any' Module nemoguardrails* Class nemoguardrails.actions.action_dispatcher.ActionDispatcher Method execute_action: (self, action_name: str, params: Dict[str, Any]) -> Tuple[Union[str, Dict[str, Any]], str] Class nemoguardrails.actions.llm.generation.LLMGenerationActions Method generate_user_intent: (self, events: List[dict], context: dict, config: nemoguardrails.rails.llm.config.RailsConfig, llm: Optional[langchain_core.language_models.llms.BaseLLM] = None, kb: Optional[nemoguardrails.kb.kb.KnowledgeBase] = None) Method generate_next_step: (self, events: List[dict], llm: Optional[langchain_core.language_models.llms.BaseLLM] = None) Method generate_bot_message: (self, events: List[dict], context: dict, llm: Optional[langchain_core.language_models.llms.BaseLLM] = None) Method generate_value: (self, instructions: str, events: List[dict], var_name: Optional[str] = None, llm: Optional[langchain_core.language_models.llms.BaseLLM] = None) Method generate_intent_steps_message: (self, events: List[dict], llm: Optional[langchain_core.language_models.llms.BaseLLM] = None, kb: Optional[nemoguardrails.kb.kb.KnowledgeBase] = None) Class nemoguardrails.kb.kb.KnowledgeBase Method search_relevant_chunks: (self, text, max_results: int = 3) Class nemoguardrails.rails.llm.llmrails.LLMRails Method generate: (self, prompt: Optional[str] = None, messages: Optional[List[dict]] = None, return_context: bool = False, options: Union[dict, nemoguardrails.rails.llm.options.GenerationOptions, NoneType] = None) Method generate_async: (self, prompt: Optional[str] = None, messages: Optional[List[dict]] = None, options: Union[dict, nemoguardrails.rails.llm.options.GenerationOptions, NoneType] = None, streaming_handler: Optional[nemoguardrails.streaming.StreamingHandler] = None, return_context: bool = False) -> Union[str, dict, nemoguardrails.rails.llm.options.GenerationResponse, Tuple[dict, dict]] Method stream_async: (self, prompt: Optional[str] = None, messages: Optional[List[dict]] = None) -> AsyncIterator[str] Method generate_events: (self, events: List[dict]) -> List[dict] Method generate_events_async: (self, events: List[dict]) -> List[dict] Method _get_events_for_messages: (self, messages: List[dict]) Module trulens_eval.* Class trulens_eval.feedback.feedback.Feedback Method __call__: (self, *args, **kwargs) -> 'Any' Class trulens_eval.tru_rails.FeedbackActions Class trulens_eval.utils.langchain.WithFeedbackFilterDocuments Method _get_relevant_documents: (self, query: str, *, run_manager) -> List[langchain_core.documents.base.Document] Method get_relevant_documents: (self, query: 'str', *, callbacks: 'Callbacks' = None, tags: 'Optional[List[str]]' = None, metadata: 'Optional[Dict[str, Any]]' = None, run_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[Document]' Method aget_relevant_documents: (self, query: 'str', *, callbacks: 'Callbacks' = None, tags: 'Optional[List[str]]' = None, metadata: 'Optional[Dict[str, Any]]' = None, run_name: 'Optional[str]' = None, **kwargs: 'Any') -> 'List[Document]' Method _aget_relevant_documents: (self, query: 'str', *, run_manager: 'AsyncCallbackManagerForRetrieverRun') -> 'List[Document]'
Instrumenting other classes/methods.¶
Additional classes and methods can be instrumented by use of the
trulens_eval.instruments.Instrument
methods and decorators. Examples of
such usage can be found in the custom app used in the custom_example.ipynb
notebook which can be found in
trulens_eval/examples/expositional/end2end_apps/custom_app/custom_app.py
. More
information about these decorators can be found in the
docs/trulens_eval/tracking/instrumentation/index.ipynb
notebook.
Inspecting instrumentation¶
The specific objects (of the above classes) and methods instrumented for a
particular app can be inspected using the App.print_instrumented
as
exemplified in the next cell. Unlike Instrument.print_instrumentation
, this
function only shows what in an app was actually instrumented.
tru_recorder.print_instrumented()
Components: TruRails (Other) at 0x2aa583d40 with path __app__ LLMRails (Custom) at 0x10464b950 with path __app__.app KnowledgeBase (Custom) at 0x2a945d5d0 with path __app__.app.kb OpenAI (Custom) at 0x2a8f61c70 with path __app__.app.llm LLMGenerationActions (Custom) at 0x29c04c990 with path __app__.app.llm_generation_actions OpenAI (Custom) at 0x2a8f61c70 with path __app__.app.llm_generation_actions.llm Methods: Object at 0x29c04c990: <function LLMGenerationActions.generate_user_intent at 0x2a898fc40> with path __app__.app.llm_generation_actions <function LLMGenerationActions.generate_next_step at 0x2a898fd80> with path __app__.app.llm_generation_actions <function LLMGenerationActions.generate_bot_message at 0x2a898fec0> with path __app__.app.llm_generation_actions <function LLMGenerationActions.generate_value at 0x2a898ff60> with path __app__.app.llm_generation_actions <function LLMGenerationActions.generate_intent_steps_message at 0x2a89b8040> with path __app__.app.llm_generation_actions Object at 0x2a945d5d0: <function KnowledgeBase.search_relevant_chunks at 0x2a898cf40> with path __app__.app.kb Object at 0x10464b950: <function LLMRails.generate at 0x2a8db7b00> with path __app__.app <function LLMRails.generate_async at 0x2a8d6ab60> with path __app__.app <function LLMRails.stream_async at 0x2a8db7880> with path __app__.app <function LLMRails.generate_events at 0x2a8df80e0> with path __app__.app <function LLMRails.generate_events_async at 0x2a8df8040> with path __app__.app <function LLMRails._get_events_for_messages at 0x2a8d234c0> with path __app__.app Object at 0x104aa42d0: <function ActionDispatcher.execute_action at 0x2a8a044a0> with path __app__.app.runtime.action_dispatcher