trulens.core.utils.stats¶

trulens.core.utils.stats ¶

Lightweight statistical helpers for TruLens.

Canonical home for paired-sample tests and confidence intervals so that trulens-core, trulens-feedback, and trulens-benchmark all share a single implementation.

Functions¶

paired_permutation_pvalue ¶

paired_permutation_pvalue(
    diffs: list[float] | ndarray,
    seed: int = 0,
    permutations: int = 10000,
) -> float

Two-sided paired sign-flip permutation p-value for mean(diffs) == 0.

Tests whether the mean of paired score differences differs significantly from zero without assuming normality, by comparing the observed absolute mean against the distribution obtained from random sign flips.

PARAMETER	DESCRIPTION
`diffs`	Paired differences, e.g. per-example score deltas between two judge configurations. TYPE: `list[float] \| ndarray`
`seed`	Seed for the permutation random number generator. TYPE: `int` DEFAULT: `0`
`permutations`	Number of sign-flip permutations to sample. TYPE: `int` DEFAULT: `10000`

RETURNS	DESCRIPTION
`float`	The two-sided p-value in `(0, 1]`. Returns `1.0` when there are
`float`	no differences or all differences are zero.

bootstrap_ci ¶

bootstrap_ci(
    diffs: ndarray,
    alpha: float = SIGNIFICANCE_ALPHA,
    n_bootstrap: int = 10000,
    seed: int = 0,
) -> tuple[float, float]

Percentile bootstrap confidence interval for the mean of diffs.

PARAMETER	DESCRIPTION
`diffs`	1-D array of paired differences. TYPE: `ndarray`
`alpha`	Significance level (default 0.05 for a 95 % CI). TYPE: `float` DEFAULT: `SIGNIFICANCE_ALPHA`
`n_bootstrap`	Number of bootstrap resamples. TYPE: `int` DEFAULT: `10000`
`seed`	RNG seed for reproducibility. TYPE: `int` DEFAULT: `0`

RETURNS	DESCRIPTION
`float`	`(ci_lower, ci_upper)` bounds. Returns `(nan, nan)` when
`float`	n < 2 (a CI is not meaningful with fewer than two observations).