trulens.benchmark.generate.generate_test_set¶

trulens.benchmark.generate.generate_test_set ¶

Classes¶

GenerateTestSet ¶

This class is responsible for generating a test set using the provided application callable.

Functions¶

init ¶

__init__(app_callable: Callable)

Initialize the GenerateTestSet class.

Parameters: app_callable (Callable): The application callable to be used for generating the test set.

generate_test_set ¶

generate_test_set(
    test_breadth: int,
    test_depth: int,
    examples: Optional[list] = None,
) -> dict

Generate a test set, optionally using few shot examples provided.

Parameters: test_breadth (int): The breadth of the test set. test_depth (int): The depth of the test set. examples (Optional[list]): An optional list of examples to guide the style of the questions.

Returns: dict: A dictionary containing the test set.

Usage example:

Instantiate GenerateTestSet with your app callable, in this case: rag_chain.invoke¶

test = GenerateTestSet(app_callable = rag_chain.invoke)

Generate the test set of a specified breadth and depth without examples¶

test_set = test.generate_test_set(test_breadth = 3, test_depth = 2)

Generate the test set of a specified breadth and depth with examples¶

examples = ["Why is it hard for AI to plan very far into the future?", "How could letting AI reflect on what went wrong help it improve in the future?"] test_set_with_examples = test.generate_test_set(test_breadth = 3, test_depth = 2, examples = examples)

trulens.benchmark.generate.generate_test_set¶