📓 Hotspots: detecting suspicious features in your evals¶

TruLens Hotspots is a tool for detecting suspicious features in your evaluation results. For instance, it can detect that a specific word in the input lowers the evaluation score much.

In [ ]:

# !pip install trulens-hotspots

# !pip install trulens-hotspots

Running Hotspots on a CSV file¶

Hotspots can be run in a simple way, without using any other TruLens features.

What you need for TruLens Hotspots is a data frame with per-sample evaluation scores (but it does matter what evaluation metric you're specific).

So let's read a CSV file.

In [ ]:

from pandas import read_csv

df = read_csv("../../tests/files/sample.csv.gz")

df

from pandas import read_csv df = read_csv("../../tests/files/sample.csv.gz") df

Now you need to define a TruLens Hotspots configuration; you're required to specify the column with the evaluation score. You can also list irrelevant columns to be skipped. If your evaluation metric is the-lower-the-better, like Mean Absolute/Square Error, you need to state that explicitly.

In [ ]:

from trulens.hotspots import HotspotsConfig
from trulens.hotspots import hotspots_as_df

hotspots_config = HotspotsConfig(
    score_column="score", skip_columns=["id"], higher_is_better=False
)

from trulens.hotspots import HotspotsConfig from trulens.hotspots import hotspots_as_df hotspots_config = HotspotsConfig( score_column="score", skip_columns=["id"], higher_is_better=False )

Running Hotspots is simple:

In [ ]:

hotspots_df = hotspots_as_df(hotspots_config, df)

hotspots_df

hotspots_df = hotspots_as_df(hotspots_config, df) hotspots_df

The task being evaluated is predicting the publication year of a short historical text. Older texts are the hardest, also some specific words make the overall score worse.