trulens.core.run¶

trulens.core.run ¶

Classes¶

ItemDiff `dataclass` ¶

Per-item score comparison between two runs for a single metric.

MetricDiff `dataclass` ¶

Aggregate comparison for a single metric across matched items.

ATTRIBUTE	DESCRIPTION
`higher_is_better`	Direction of the metric. `True` means a positive delta is an improvement; `False` means a negative delta is an improvement. TYPE: `bool`

RunDiff `dataclass` ¶

Result of comparing two runs.

.. note:: Experimental -- this class is part of the run-comparison API introduced in #2629. Its shape may change as the leaderboard work (#2619) lands.

Returned by :meth:Run.compare and :func:compare_runs.

ATTRIBUTE	DESCRIPTION
`run_a_name`	Name of the baseline run (`self` in `run_a.compare(run_b)`). TYPE: `str`
`run_b_name`	Name of the candidate run. TYPE: `str`
`metrics`	Per-metric comparison results keyed by metric name. TYPE: `Dict[str, MetricDiff]`

Functions¶

summary ¶

summary() -> DataFrame

Return a one-row-per-metric summary DataFrame.

items_df ¶

items_df(metric_name: str) -> DataFrame

Return per-item details for a given metric as a DataFrame.

Run ¶

Bases: BaseModel

Attributes¶

model_config `class-attribute` ¶

model_config: ConfigDict = ConfigDict(
    arbitrary_types_allowed=True, extra="ignore"
)

Run class for managing run state / attributes in the SDK client.

This model is meant to be used and accessed through methods like describe() (which uses the underlying RunDao) to obtain the run metadata.

Functions¶

describe ¶

describe() -> dict

Retrieve the metadata of the Run object.

delete ¶

delete() -> None

Delete the run by its name and object name.

start ¶

start(input_df: Optional[DataFrame] = None)

Start the run by invoking the main method of the user's app with the input data.

PARAMETER	DESCRIPTION
`input_df`	Optional input dataframe. If not provided, data is fetched from the source configured in the run's `RunConfig`. TYPE: `Optional[DataFrame]` DEFAULT: `None`

Note

After the app finishes executing, this method kicks off a Snowflake server-side ingestion stored procedure (start_ingestion_query) asynchronously. The method returns before that ingestion is complete. Calling compute_metrics() immediately after start() may therefore find zero events and silently compute nothing.

Wait until run.get_status() returns RunStatus.INVOCATION_COMPLETED (or another terminal invocation state) before calling compute_metrics(). RecordIngestMode.IMMEDIATE only controls the client-side OTEL span exporter flush and does not guarantee that the server-side ingestion sproc has finished.

compute_metrics ¶

compute_metrics(
    metrics: List[
        Union[str, "metric_module.Metric", MetricConfig]
    ],
) -> str

Compute metrics for the run.

PARAMETER	DESCRIPTION
`metrics`	List of metrics to compute. Each entry can be: - A `str` — name of a server-side (Snowflake-hosted) metric. - A :class:`~trulens.core.metric.metric.Metric` — a client-side custom metric defined with the current API. - A :class:`~trulens.core.feedback.custom_metric.MetricConfig` — deprecated; use `Metric` instead. TYPE: `List[Union[str, 'metric_module.Metric', MetricConfig]]`

RETURNS	DESCRIPTION
`str`	Status message indicating computation progress.

Note

Snowflake record ingestion triggered by run.start() is asynchronous. Call this method only after run.get_status() indicates ingestion has completed (e.g. RunStatus.INVOCATION_COMPLETED). Calling it too soon after start() may result in 0 events being found and no metrics being computed.

get_records ¶

get_records(
    record_ids: Optional[List[str]] = None,
    offset: Optional[int] = None,
    limit: Optional[int] = None,
) -> DataFrame

A wrapper API around get_records_and_feedback to retrieve and display overview of records from event table of the run. It aggregates summary information of records into a single DataFrame.

PARAMETER	DESCRIPTION
`record_ids`	Optional list of record IDs to filter by. Defaults to None. TYPE: `Optional[List[str]]` DEFAULT: `None`
`offset`	Record row offset. TYPE: `Optional[int]` DEFAULT: `None`
`limit`	Limit on the number of records to return. TYPE: `Optional[int]` DEFAULT: `None`

RETURNS	DESCRIPTION
`DataFrame`	A DataFrame with the overview of records.

get_record_details ¶

get_record_details(
    record_ids: Optional[List[str]] = None,
    offset: Optional[int] = None,
    limit: Optional[int] = None,
) -> DataFrame

A wrapper API around get_records_and_feedback to retrieve records from event table of the run.

PARAMETER	DESCRIPTION
`record_ids`	Optional list of record IDs to filter by. Defaults to None. TYPE: `Optional[List[str]]` DEFAULT: `None`
`offset`	Record row offset. TYPE: `Optional[int]` DEFAULT: `None`
`limit`	Limit on the number of records to return. TYPE: `Optional[int]` DEFAULT: `None`

RETURNS	DESCRIPTION
`DataFrame`	A DataFrame with the details of records.

compare ¶

compare(
    other: Run,
    tolerance: float = 0.0,
    metric_directions: Optional[Dict[str, bool]] = None,
) -> RunDiff

Compare this run (baseline) against other (candidate).

Records are matched by their input column. For each shared metric the method computes per-item deltas, flags regressions, and returns an aggregate delta with a 95 % confidence interval and a permutation p-value.

When metric_directions is None (the default) the method attempts to look up higher_is_better from the feedback definitions stored by the session. If a metric's direction cannot be resolved it defaults to True and a warning is logged once.

PARAMETER	DESCRIPTION
`other`	The candidate run to compare against. TYPE: `Run`
`tolerance`	Minimum absolute score drop to flag an item as regressed. Defaults to `0.0`. TYPE: `float` DEFAULT: `0.0`
`metric_directions`	Optional override mapping metric name to `higher_is_better`. When provided this takes precedence over the stored feedback definitions. TYPE: `Optional[Dict[str, bool]]` DEFAULT: `None`

RETURNS	DESCRIPTION
`A`	class:`RunDiff` with per-metric comparison results. TYPE: `RunDiff`

RAISES	DESCRIPTION
`ValueError`	If the two runs share no inputs or no metrics.

update ¶

update(
    description: Optional[str] = None,
    label: Optional[str] = None,
)

Only description and label are allowed to be updated at the moment.

from_metadata_df `classmethod` ¶

from_metadata_df(
    metadata_df: DataFrame, extra: Dict[str, Any]
) -> Run

Create a Run instance from a metadata DataFrame returned by the DAO, and enrich it with additional fields (which are not persisted on the server).

PARAMETER	DESCRIPTION
`metadata_df`	A pandas DataFrame containing run metadata. We assume the first row contains a JSON string in its first cell. TYPE: `DataFrame`
`extra`	A dictionary of extra fields to add, such as: { "app": , "main_method_name": , "run_dao": , "object_name":

RETURNS	DESCRIPTION
`Run`	A validated Run instance.

Functions¶

validate_dataset_spec ¶

validate_dataset_spec(
    dataset_spec: Dict[str, str],
) -> Dict[str, str]

Validates and normalizes the dataset column specification to ensure it contains only currently supported span attributes and that the keys are in the correct format.

PARAMETER	DESCRIPTION
`dataset_spec`	The user-provided dictionary with column names. TYPE: `Dict[str, str]`

RETURNS	DESCRIPTION
`Dict[str, str]`	A validated and normalized dictionary.

RAISES	DESCRIPTION
`ValueError`	If any invalid field is present.

compare_runs ¶

compare_runs(
    run_a: Run,
    run_b: Run,
    tolerance: float = 0.0,
    metric_directions: Optional[Dict[str, bool]] = None,
) -> RunDiff

Compare two runs over the same inputs and detect regressions.

Records are matched by their normalised input value. When the same input appears more than once in a run (retries, synthetic duplicates, etc.) records are paired positionally within each duplicate group -- the k-th occurrence in run A (ordered by record_id) matches the k-th occurrence in run B. The pairing is stable but arbitrary; aggregate stats are the trustworthy output for duplicated inputs. Unmatched extras are dropped with a warning.

For every shared metric column the function computes:

per-item delta -- score_b - score_a (positive means candidate scored higher).
regression flag -- True when the score got worse beyond tolerance. Higher-is-better metrics regress when delta < -tolerance; lower-is-better metrics regress when delta > tolerance.
aggregate delta -- mean of the per-item deltas with a 95 % bootstrap confidence interval and a two-sided permutation p-value. These are complementary resampling methods that may disagree marginally near the significance boundary; this is expected.

PARAMETER	DESCRIPTION
`run_a`	Baseline run. TYPE: `Run`
`run_b`	Candidate run. TYPE: `Run`
`tolerance`	Minimum absolute score change to flag an item as regressed. Defaults to `0.0`. TYPE: `float` DEFAULT: `0.0`
`metric_directions`	Mapping of metric name to `higher_is_better` (`True` = higher is better, `False` = lower is better). Metrics not listed default to `True` and a warning is logged. TYPE: `Optional[Dict[str, bool]]` DEFAULT: `None`

RETURNS	DESCRIPTION
`A`	class:`RunDiff` with per-metric comparison results. TYPE: `RunDiff`

RAISES	DESCRIPTION
`ValueError`	If the runs share no inputs or no metric columns.

Example::

run_a = Run(app_v1, ...).start()
run_b = Run(app_v2, ...).start()
diff = compare_runs(run_a, run_b)
print(diff.summary())

trulens.core.run¶

trulens.core.run ¶

Classes¶

ItemDiff dataclass ¶

MetricDiff dataclass ¶

RunDiff dataclass ¶

Functions¶

summary ¶

items_df ¶

Run ¶

Attributes¶

model_config class-attribute ¶

Functions¶

describe ¶

delete ¶

start ¶

compute_metrics ¶

get_records ¶

get_record_details ¶

compare ¶

update ¶

from_metadata_df classmethod ¶

Functions¶

validate_dataset_spec ¶

compare_runs ¶

ItemDiff `dataclass` ¶

MetricDiff `dataclass` ¶

RunDiff `dataclass` ¶

model_config `class-attribute` ¶

from_metadata_df `classmethod` ¶