Skip to content

Visualization Methods

trulens_explain.trulens.visualizations

One clear use case for measuring attributions is for human consumption. In order to be fully leveraged by humans, explanations need to be interpretable — a large vector of numbers doesn’t in general make us more confident we understand what a network is doing. We therefore view an explanation as comprised of both an attribution measurement and an interpretation of what the attribution values represent.

One obvious way to interpret attributions, particularly in the image domain, is via visualization. This module provides several visualization methods for interpreting attributions as images.

Classes

Tiler

Bases: object

Used to tile batched images or attributions.

Functions
tile
tile(a: ndarray) -> ndarray

Tiles the given array into a grid that is as square as possible.

PARAMETER DESCRIPTION
a

An array of 4D batched image data.

TYPE: ndarray

RETURNS DESCRIPTION
ndarray

A tiled array of the images from a. The resulting array has rank

ndarray

3 for color images, and 2 for grayscale images (the batch dimension

ndarray

is removed, as well as the channel dimension for grayscale images).

ndarray

The resulting array has its color channel dimension ordered last to

ndarray

fit the requirements of the matplotlib library.

Visualizer

Bases: object

Visualizes attributions directly as a color image. Intended particularly for use with input-attributions.

This can also be used for viewing images (rather than attributions).

Functions
__init__
__init__(combine_channels: bool = False, normalization_type: str = None, blur: float = 0.0, cmap: Colormap = None)

Configures the default parameters for the __call__ method (these can be overridden by passing in values to __call__).

PARAMETER DESCRIPTION
combine_channels

If True, the attributions will be averaged across the channel dimension, resulting in a 1-channel attribution map.

TYPE: bool DEFAULT: False

normalization_type

Specifies one of the following configurations for normalizing the attributions (each item is normalized separately):

  • 'unsigned_max': normalizes the attributions to the range [-1, 1] by dividing the attributions by the maximum absolute attribution value.
  • 'unsigned_max_positive_centered': same as above, but scales the values to the range [0, 1], with negative scores less than 0.5 and positive scores greater than 0.5.
  • 'magnitude_max': takes the absolute value of the attributions, then normalizes the attributions to the range [0, 1] by dividing by the maximum absolute attribution value.
  • 'magnitude_sum': takes the absolute value of the attributions, then scales them such that they sum to 1. If this option is used, each channel is normalized separately, such that each channel sums to 1.
  • 'signed_max': normalizes the attributions to the range [-1, 1] by dividing the positive values by the maximum positive attribution value and the negative values by the minimum negative attribution value.
  • 'signed_max_positive_centered': same as above, but scales the values to the range [0, 1], with negative scores less than 0.5 and positive scores greater than 0.5.
  • 'signed_sum': scales the positive attributions such that they sum to 1 and the negative attributions such that they scale to -1. If this option is used, each channel is normalized separately.
  • '01': normalizes the attributions to the range [0, 1] by subtracting the minimum attribution value then dividing by the maximum attribution value.
  • 'unnormalized': leaves the attributions unaffected.

If None, either 'unsigned_max' (for single-channel data) or 'unsigned_max_positive_centered' (for multi-channel data) is used.

TYPE: str DEFAULT: None

blur

Gives the radius of a Gaussian blur to be applied to the attributions before visualizing. This can be used to help focus on salient regions rather than specific salient pixels.

TYPE: float DEFAULT: 0.0

cmap

matplotlib.colors.Colormap | str, optional Colormap or name of a Colormap to use for the visualization. If None, the colormap will be chosen based on the normalization type. This argument is only used for single-channel data (including when combine_channels is True).

TYPE: Colormap DEFAULT: None

__call__
__call__(attributions, output_file=None, imshow=True, fig=None, return_tiled=False, combine_channels=None, normalization_type=None, blur=None, cmap=None) -> ndarray

Visualizes the given attributions.

PARAMETER DESCRIPTION
attributions

A np.ndarray containing the attributions to be visualized.

output_file

File name to save the visualization image to. If None, no image will be saved, but the figure can still be displayed.

DEFAULT: None

imshow

If true, a the visualization will be displayed. Otherwise the figure will not be displayed, but the figure can still be saved.

DEFAULT: True

fig

The pyplot figure to display the visualization in. If None, a new figure will be created.

DEFAULT: None

return_tiled

If true, the returned array will be in the same shape as the visualization, with no batch dimension and the samples in the batch tiled along the width and height dimensions. If false, the returned array will be reshaped to match attributions.

DEFAULT: False

combine_channels

If True, the attributions will be averaged across the channel dimension, resulting in a 1-channel attribution map. If None, defaults to the value supplied to the constructor.

DEFAULT: None

normalization_type

Specifies one of the following configurations for normalizing the attributions (each item is normalized separately):

  • 'unsigned_max': normalizes the attributions to the range [-1, 1] by dividing the attributions by the maximum absolute attribution value.
  • 'unsigned_max_positive_centered': same as above, but scales the values to the range [0, 1], with negative scores less than 0.5 and positive scores greater than 0.5.
  • 'magnitude_max': takes the absolute value of the attributions, then normalizes the attributions to the range [0, 1] by dividing by the maximum absolute attribution value.
  • 'magnitude_sum': takes the absolute value of the attributions, then scales them such that they sum to 1. If this option is used, each channel is normalized separately, such that each channel sums to 1.
  • 'signed_max': normalizes the attributions to the range [-1, 1] by dividing the positive values by the maximum positive attribution value and the negative values by the minimum negative attribution value.
  • 'signed_max_positive_centered': same as above, but scales the values to the range [0, 1], with negative scores less than 0.5 and positive scores greater than 0.5.
  • 'signed_sum': scales the positive attributions such that they sum to 1 and the negative attributions such that they scale to -1. If this option is used, each channel is normalized separately.
  • '01': normalizes the attributions to the range [0, 1] by subtracting the minimum attribution value then dividing by the maximum attribution value.
  • 'unnormalized': leaves the attributions unaffected.

If None, defaults to the value supplied to the constructor.

DEFAULT: None

blur

Gives the radius of a Gaussian blur to be applied to the attributions before visualizing. This can be used to help focus on salient regions rather than specific salient pixels. If None, defaults to the value supplied to the constructor.

DEFAULT: None

cmap

matplotlib.colors.Colormap | str, optional Colormap or name of a Colormap to use for the visualization. If None, defaults to the value supplied to the constructor.

DEFAULT: None

RETURNS DESCRIPTION
ndarray

A np.ndarray array of the numerical representation of the

ndarray

attributions as modified for the visualization. This includes

ndarray

normalization, blurring, etc.

HeatmapVisualizer

Bases: Visualizer

Visualizes attributions by overlaying an attribution heatmap over the original image, similar to how GradCAM visualizes attributions.

Functions
__init__
__init__(overlay_opacity=0.5, normalization_type=None, blur=10.0, cmap='jet')

Configures the default parameters for the __call__ method (these can be overridden by passing in values to __call__).

PARAMETER DESCRIPTION
overlay_opacity

float Value in the range [0, 1] specifying the opacity for the heatmap overlay.

DEFAULT: 0.5

normalization_type

Specifies one of the following configurations for normalizing the attributions (each item is normalized separately):

  • 'unsigned_max': normalizes the attributions to the range [-1, 1] by dividing the attributions by the maximum absolute attribution value.
  • 'unsigned_max_positive_centered': same as above, but scales the values to the range [0, 1], with negative scores less than 0.5 and positive scores greater than 0.5.
  • 'magnitude_max': takes the absolute value of the attributions, then normalizes the attributions to the range [0, 1] by dividing by the maximum absolute attribution value.
  • 'magnitude_sum': takes the absolute value of the attributions, then scales them such that they sum to 1. If this option is used, each channel is normalized separately, such that each channel sums to 1.
  • 'signed_max': normalizes the attributions to the range [-1, 1] by dividing the positive values by the maximum positive attribution value and the negative values by the minimum negative attribution value.
  • 'signed_max_positive_centered': same as above, but scales the values to the range [0, 1], with negative scores less than 0.5 and positive scores greater than 0.5.
  • 'signed_sum': scales the positive attributions such that they sum to 1 and the negative attributions such that they scale to -1. If this option is used, each channel is normalized separately.
  • '01': normalizes the attributions to the range [0, 1] by subtracting the minimum attribution value then dividing by the maximum attribution value.
  • 'unnormalized': leaves the attributions unaffected.

If None, either 'unsigned_max' (for single-channel data) or 'unsigned_max_positive_centered' (for multi-channel data) is used.

DEFAULT: None

blur

Gives the radius of a Gaussian blur to be applied to the attributions before visualizing. This can be used to help focus on salient regions rather than specific salient pixels.

DEFAULT: 10.0

cmap

matplotlib.colors.Colormap | str, optional Colormap or name of a Colormap to use for the visualization. If None, the colormap will be chosen based on the normalization type. This argument is only used for single-channel data (including when combine_channels is True).

DEFAULT: 'jet'

__call__
__call__(attributions, x, output_file=None, imshow=True, fig=None, return_tiled=False, overlay_opacity=None, normalization_type=None, blur=None, cmap=None) -> ndarray

Visualizes the given attributions by overlaying an attribution heatmap over the given image.

PARAMETER DESCRIPTION
attributions

A np.ndarray containing the attributions to be visualized.

x

A np.ndarray of items in the same shape as attributions corresponding to the records explained by the given attributions. The visualization will be superimposed onto the corresponding set of records.

output_file

File name to save the visualization image to. If None, no image will be saved, but the figure can still be displayed.

DEFAULT: None

imshow

If true, a the visualization will be displayed. Otherwise the figure will not be displayed, but the figure can still be saved.

DEFAULT: True

fig

The pyplot figure to display the visualization in. If None, a new figure will be created.

DEFAULT: None

return_tiled

If true, the returned array will be in the same shape as the visualization, with no batch dimension and the samples in the batch tiled along the width and height dimensions. If false, the returned array will be reshaped to match attributions.

DEFAULT: False

overlay_opacity

float Value in the range [0, 1] specifying the opacity for the heatmap overlay. If None, defaults to the value supplied to the constructor.

DEFAULT: None

normalization_type

Specifies one of the following configurations for normalizing the attributions (each item is normalized separately):

  • 'unsigned_max': normalizes the attributions to the range [-1, 1] by dividing the attributions by the maximum absolute attribution value.
  • 'unsigned_max_positive_centered': same as above, but scales the values to the range [0, 1], with negative scores less than 0.5 and positive scores greater than 0.5.
  • 'magnitude_max': takes the absolute value of the attributions, then normalizes the attributions to the range [0, 1] by dividing by the maximum absolute attribution value.
  • 'magnitude_sum': takes the absolute value of the attributions, then scales them such that they sum to 1. If this option is used, each channel is normalized separately, such that each channel sums to 1.
  • 'signed_max': normalizes the attributions to the range [-1, 1] by dividing the positive values by the maximum positive attribution value and the negative values by the minimum negative attribution value.
  • 'signed_max_positive_centered': same as above, but scales the values to the range [0, 1], with negative scores less than 0.5 and positive scores greater than 0.5.
  • 'signed_sum': scales the positive attributions such that they sum to 1 and the negative attributions such that they scale to -1. If this option is used, each channel is normalized separately.
  • '01': normalizes the attributions to the range [0, 1] by subtracting the minimum attribution value then dividing by the maximum attribution value.
  • 'unnormalized': leaves the attributions unaffected.

If None, defaults to the value supplied to the constructor.

DEFAULT: None

blur

Gives the radius of a Gaussian blur to be applied to the attributions before visualizing. This can be used to help focus on salient regions rather than specific salient pixels. If None, defaults to the value supplied to the constructor.

DEFAULT: None

cmap

matplotlib.colors.Colormap | str, optional Colormap or name of a Colormap to use for the visualization. If None, defaults to the value supplied to the constructor.

DEFAULT: None

RETURNS DESCRIPTION
ndarray

A np.ndarray array of the numerical representation of the

ndarray

attributions as modified for the visualization. This includes

ndarray

normalization, blurring, etc.

MaskVisualizer

Bases: object

Visualizes attributions by masking the original image to highlight the regions with influence above a given threshold percentile. Intended particularly for use with input-attributions.

Functions
__init__
__init__(blur=5.0, threshold=0.5, masked_opacity=0.2, combine_channels=True, use_attr_as_opacity=False, positive_only=True)

Configures the default parameters for the __call__ method (these can be overridden by passing in values to __call__).

PARAMETER DESCRIPTION
blur

Gives the radius of a Gaussian blur to be applied to the attributions before visualizing. This can be used to help focus on salient regions rather than specific salient pixels.

DEFAULT: 5.0

threshold

Value in the range [0, 1]. Attribution values at or below the percentile given by threshold (after normalization, blurring, etc.) will be masked.

DEFAULT: 0.5

masked_opacity

Value in the range [0, 1] specifying the opacity for the parts of the image that are masked.

DEFAULT: 0.2

combine_channels

If True, the attributions will be averaged across the channel dimension, resulting in a 1-channel attribution map.

DEFAULT: True

use_attr_as_opacity

If True, instead of using threshold and masked_opacity, the opacity of each pixel is given by the 0-1-normalized attribution value.

DEFAULT: False

positive_only

If True, only pixels with positive attribution will be unmasked (or given nonzero opacity when use_attr_as_opacity is true).

DEFAULT: True

ChannelMaskVisualizer

Bases: object

Uses internal influence to visualize the pixels that are most salient towards a particular internal channel or neuron.

Functions
__init__
__init__(model, layer, channel, channel_axis=None, agg_fn=None, doi=None, blur=None, threshold=0.5, masked_opacity=0.2, combine_channels: bool = True, use_attr_as_opacity=None, positive_only=None)

Configures the default parameters for the __call__ method (these can be overridden by passing in values to __call__).

PARAMETER DESCRIPTION
model

The wrapped model whose channel we're visualizing.

layer

The identifier (either index or name) of the layer in which the channel we're visualizing resides.

channel

Index of the channel (for convolutional layers) or internal neuron (for fully-connected layers) that we'd like to visualize.

channel_axis

If different from the channel axis specified by the backend, the supplied channel_axis will be used if operating on a convolutional layer with 4-D image format.

DEFAULT: None

agg_fn

Function with which to aggregate the remaining dimensions (except the batch dimension) in order to get a single scalar value for each channel; If None, a sum over each neuron in the channel will be taken. This argument is not used when the channels are scalars, e.g., for dense layers.

DEFAULT: None

doi

The distribution of interest to use when computing the input attributions towards the specified channel. If None, PointDoI will be used.

DEFAULT: None

blur

Gives the radius of a Gaussian blur to be applied to the attributions before visualizing. This can be used to help focus on salient regions rather than specific salient pixels.

DEFAULT: None

threshold

Value in the range [0, 1]. Attribution values at or below the percentile given by threshold (after normalization, blurring, etc.) will be masked.

DEFAULT: 0.5

masked_opacity

Value in the range [0, 1] specifying the opacity for the parts of the image that are masked.

DEFAULT: 0.2

combine_channels

If True, the attributions will be averaged across the channel dimension, resulting in a 1-channel attribution map.

TYPE: bool DEFAULT: True

use_attr_as_opacity

If True, instead of using threshold and masked_opacity, the opacity of each pixel is given by the 0-1-normalized attribution value.

DEFAULT: None

positive_only

If True, only pixels with positive attribution will be unmasked (or given nonzero opacity when use_attr_as_opacity is true).

DEFAULT: None

__call__
__call__(x, x_preprocessed=None, output_file=None, blur=None, threshold=None, masked_opacity=None, combine_channels=None)

Visualizes the given attributions by overlaying an attribution heatmap over the given image.

Parameters

attributions : numpy.ndarray The attributions to visualize. Expected to be in 4-D image format.

numpy.ndarray

The original image(s) over which the attributions are calculated. Must be the same shape as expected by the model used with this visualizer.

numpy.ndarray, optional

If the model requires a preprocessed input (e.g., with the mean subtracted) that is different from how the image should be visualized, x_preprocessed should be specified. In this case x will be used for visualization, and x_preprocessed will be passed to the model when calculating attributions. Must be the same shape as x.

str, optional

If specified, the resulting visualization will be saved to a file with the name given by output_file.

float, optional

If specified, gives the radius of a Gaussian blur to be applied to the attributions before visualizing. This can be used to help focus on salient regions rather than specific salient pixels. If None, defaults to the value supplied to the constructor. Default None.

float

Value in the range [0, 1]. Attribution values at or below the percentile given by threshold will be masked. If None, defaults to the value supplied to the constructor. Default None.

float

Value in the range [0, 1] specifying the opacity for the parts of the image that are masked. Default 0.2. If None, defaults to the value supplied to the constructor. Default None.

bool

If True, the attributions will be averaged across the channel dimension, resulting in a 1-channel attribution map. If None, defaults to the value supplied to the constructor. Default None.

Output

Bases: ABC

Base class for visualization output formats.

PlainText

Bases: Output

Plain text visualization output format.

HTML

Bases: Output

HTML visualization output format.

IPython

Bases: HTML

Interactive python visualization output format.

NLP

Bases: object

NLP Visualization tools.

Functions
__init__
__init__(wrapper: ModelWrapper, output: Optional[Output] = None, labels: Optional[Iterable[str]] = None, tokenize: Optional[Callable[[TextBatch], ModelInputs]] = None, decode: Optional[Callable[[Tensor], str]] = None, input_accessor: Optional[Callable[[ModelInputs], Iterable[Tensor]]] = None, output_accessor: Optional[Callable[[ModelOutput], Iterable[Tensor]]] = None, attr_aggregate: Optional[Callable[[Tensor], Tensor]] = None, hidden_tokens: Optional[Set[int]] = set())

Initializate NLP visualization tools for a given environment.

PARAMETER DESCRIPTION
wrapper

ModelWrapper The wrapped model whose channel we're visualizing.

TYPE: ModelWrapper

output

Output, optional Visualization output format. Defaults to PlainText unless ipython is detected and in which case defaults to IPython format.

TYPE: Optional[Output] DEFAULT: None

labels

Iterable[str], optional Names of prediction classes for classification models.

TYPE: Optional[Iterable[str]] DEFAULT: None

tokenize

Callable[[TextBatch], ModelInput], optional Method to tokenize an instance.

TYPE: Optional[Callable[[TextBatch], ModelInputs]] DEFAULT: None

decode

Callable[[Tensor], str], optional Method to invert/decode the tokenization.

TYPE: Optional[Callable[[Tensor], str]] DEFAULT: None

input_accessor

Callable[[ModelInputs], Iterable[Tensor]], optional Method to extract input/token ids from model inputs (tokenize output) if needed.

TYPE: Optional[Callable[[ModelInputs], Iterable[Tensor]]] DEFAULT: None

output_accessor

Callable[[ModelOutput], Iterable[Tensor]], optional Method to extract outout logits from output structures if needed.

TYPE: Optional[Callable[[ModelOutput], Iterable[Tensor]]] DEFAULT: None

attr_aggregate

Callable[[Tensor], Tensor], optional Method to aggregate attribution for embedding into a single value. Defaults to sum.

TYPE: Optional[Callable[[Tensor], Tensor]] DEFAULT: None

hidden_tokens

Set[int], optional For token-based visualizations, which tokens to hide.

TYPE: Optional[Set[int]] DEFAULT: set()

token_attribution
token_attribution(texts: Iterable[str], attr: AttributionMethod)

Visualize a token-based input attribution on given texts inputs via the attribution method attr.

PARAMETER DESCRIPTION
texts

Iterable[str] The input texts to visualize.

TYPE: Iterable[str]

attr

AttributionMethod The attribution method to generate the token importances with.

TYPE: AttributionMethod

ANY DESCRIPTION

The visualization in the format specified by this class's output parameter.