trulens.core.feedback.custom_metric¶

trulens.core.feedback.custom_metric ¶

Client-side custom metrics functionality.

Classes¶

MetricConfig ¶

Configuration for a custom metric including implementation and span mapping.

This class defines a complete metric configuration that includes the metric function implementation and how its arguments should be extracted from OTEL spans.

Key Concepts: - metric_name: Unique semantic identifier for this specific usage of the metric (e.g., "text2sql_accuracy_v1", "relevance_for_qa_task") - metric_type: Implementation identifier of the underlying metric function (e.g., "text2sql", "accuracy", "relevance")

This distinction allows the same metric implementation to be used multiple times with different configurations and names within the same application.

Functions¶

init ¶

__init__(
    metric_name: str,
    metric_implementation: Callable,
    metric_type: Optional[str] = None,
    selectors: Optional[Dict[str, Selector]] = None,
    computation_type: str = "client",
    higher_is_better: bool = True,
    description: Optional[str] = None,
)

Initialize a metric configuration.

PARAMETER	DESCRIPTION
`metric_name`	Unique semantic identifier for this specific metric usage (e.g., "text2sql_accuracy_v1", "custom_relevance_for_qa") TYPE: `str`
`metric_implementation`	The metric function to execute TYPE: `Callable`
`metric_type`	Implementation identifier of the custom metric (e.g., "text2sql", "accuracy", "relevance"). If not provided, defaults to the function name. TYPE: `Optional[str]` DEFAULT: `None`
`selectors`	Dictionary mapping parameter names to Selectors TYPE: `Optional[Dict[str, Selector]]` DEFAULT: `None`
`computation_type`	Where to compute ("client" or "server") TYPE: `str` DEFAULT: `'client'`
`higher_is_better`	Whether higher scores are better TYPE: `bool` DEFAULT: `True`
`description`	Optional description of the metric TYPE: `Optional[str]` DEFAULT: `None`

validate_selectors ¶

validate_selectors() -> None

Validate that selectors match the function signature.

Only checks required parameters (those without default values).

RAISES	DESCRIPTION
`ValueError`	If selectors don't match function parameters

create_feedback_definition ¶

create_feedback_definition() -> Feedback

Create a Feedback instance from this metric configuration.

RETURNS	DESCRIPTION
`Feedback`	A Feedback instance configured for this metric

RAISES	DESCRIPTION
`ValueError`	If selectors don't match function parameters

to_dict ¶

to_dict() -> Dict[str, Any]

Convert the metric config to a dictionary.

RETURNS	DESCRIPTION
`Dict[str, Any]`	Dictionary representation of the config

trulens.core.feedback.custom_metric¶