Tru Rails for NeMo Guardrails

trulens_eval.tru_rails.TruRails

Bases: App

Recorder for apps defined using NeMo Guardrails.

PARAMETER	DESCRIPTION
app	A NeMo Guardrails application. TYPE: LLMRails

Functions

main_output

main_output(func: Callable, sig: Signature, bindings: BoundArguments, ret: Any) -> JSON

Determine the main out string for the given function func with signature sig after it is called with the given bindings and has returned ret.

main_input

main_input(func: Callable, sig: Signature, bindings: BoundArguments) -> JSON

Determine the main input string for the given function func with signature sig after it is called with the given bindings and has returned ret.

select_context classmethod

select_context(app: Optional[LLMRails] = None) -> Lens

Get the path to the context in the query output.

trulens_eval.tru_rails.RailsActionSelect

Bases: Select

Selector shorthands for NeMo Guardrails apps when used for evaluating feedback in actions.

These should not be used for feedback functions given to TruRails but instead for selectors in the FeedbackActions action invoked from with a rails app.

Attributes

Action class-attribute instance-attribute

Action = action

Selector for action call parameters.

Events class-attribute instance-attribute

Events = events

Selector for events in action call parameters.

Context class-attribute instance-attribute

Context = context

Selector for context in action call parameters.

Warning

This is not the same "context" as in RAG triad. This is a parameter to rails actions that stores context of the rails app execution.

LLM class-attribute instance-attribute

LLM = llm

Selector for the language model in action call parameters.

Config class-attribute instance-attribute

Config = config

Selector for the configuration in action call parameters.

RetrievalContexts class-attribute instance-attribute

RetrievalContexts = relevant_chunks_sep

Selector for the retrieved contexts chunks returned from a KB search.

Equivalent to $relevant_chunks_sep in colang.

UserMessage class-attribute instance-attribute

UserMessage = user_message

Selector for the user message.

Equivalent to $user_message in colang.

BotMessage class-attribute instance-attribute

BotMessage = bot_message

Selector for the bot message.

Equivalent to $bot_message in colang.

LastUserMessage class-attribute instance-attribute

LastUserMessage = last_user_message

Selector for the last user message.

Equivalent to $last_user_message in colang.

LastBotMessage class-attribute instance-attribute

LastBotMessage = last_bot_message

Selector for the last bot message.

Equivalent to $last_bot_message in colang.

trulens_eval.tru_rails.FeedbackActions

Feedback action action for NeMo Guardrails apps.

See docstring of method feedback.

Functions

register_feedback_functions staticmethod

register_feedback_functions(*args: Tuple[Feedback, ...], **kwargs: Dict[str, Feedback])

Register one or more feedback functions to use in rails feedback action.

All keyword arguments indicate the key as the keyword. All positional arguments use the feedback name as the key.

action_of_feedback staticmethod

action_of_feedback(feedback_instance: Feedback, verbose: bool = False) -> Callable

Create a custom rails action for the given feedback function.

PARAMETER	DESCRIPTION
feedback_instance	A feedback function to register as an action. TYPE: Feedback
verbose	Print out info on invocation upon invocation. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
Callable	A custom action that will run the feedback function. The name is the same as the feedback function's name.

feedback_action async staticmethod

feedback_action(events: Optional[List[Dict]] = None, context: Optional[Dict] = None, llm: Optional[BaseLanguageModel] = None, config: Optional[RailsConfig] = None, function: Optional[str] = None, selectors: Optional[Dict[str, Union[str, Lens]]] = None, verbose: bool = False) -> ActionResult

Run the specified feedback function from trulens_eval.

To use this action, it needs to be registered with your rails app and feedback functions themselves need to be registered with this function. The name under which this action is registered for rails is feedback.

Usage

rails: LLMRails = ... # your app
language_match: Feedback = Feedback(...) # your feedback function

# First we register some feedback functions with the custom action:
FeedbackAction.register_feedback_functions(language_match)

# Can also use kwargs expansion from dict like produced by rag_triad:
# FeedbackAction.register_feedback_functions(**rag_triad(...))

# Then the feedback method needs to be registered with the rails app:
rails.register_action(FeedbackAction.feedback)

PARAMETER	DESCRIPTION
events	See Action parameters. TYPE: Optional[List[Dict]] DEFAULT: None
context	See Action parameters. TYPE: Optional[Dict] DEFAULT: None
llm	See Action parameters. TYPE: Optional[BaseLanguageModel] DEFAULT: None
config	See Action parameters. TYPE: Optional[RailsConfig] DEFAULT: None
function	Name of the feedback function to run. TYPE: Optional[str] DEFAULT: None
selectors	Selectors for the function. Can be provided either as strings to be parsed into lenses or lenses themselves. TYPE: Optional[Dict[str, Union[str, Lens]]] DEFAULT: None
verbose	Print the values of the selectors before running feedback and print the result after running feedback. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
ActionResult	An action result containing the result of the feedback. TYPE: ActionResult

Example

define subflow check language match
    $result = execute feedback(\
        function="language_match",\
        selectors={\
        "text1":"action.context.last_user_message",\
        "text2":"action.context.bot_message"\
        }\
    )
    if $result < 0.8
        bot inform language mismatch
        stop

trulens_eval.tru_rails.RailsInstrument

Bases: Instrument

Instrumentation specification for NeMo Guardrails apps.

Classes

Default

Default instrumentation specification.

Attributes

MODULES class-attribute instance-attribute

MODULES = union(MODULES)

Modules to instrument by name prefix.

Note that NeMo Guardrails uses LangChain internally for some things.

CLASSES class-attribute instance-attribute

CLASSES = lambda: union(CLASSES())

Instrument only these classes.

METHODS class-attribute instance-attribute

METHODS: Dict[str, ClassFilter] = dict_set_with_multikey(dict(METHODS), {'execute_action': ActionDispatcher, ('generate', 'generate_async', 'stream_async', 'generate_events', 'generate_events_async', '_get_events_for_messages'): LLMRails, 'search_relevant_chunks': KnowledgeBase, ('generate_user_intent', 'generate_next_step', 'generate_bot_message', 'generate_value', 'generate_intent_steps_message'): LLMGenerationActions, 'feedback': FeedbackActions})

Instrument only methods with these names and of these classes.