trulens.feedback.v2.feedback

trulens.feedback.v2.feedback

Classes

Feedback

Bases: BaseModel

Base class for feedback functions.

Criteria

Bases: str, Enum

A Criteria to evaluate.

Relevance

Bases: Semantics

This evaluates the relevance of the LLM response to the given text by LLM prompting.

Relevance is available for any LLM provider.

OutputSpace

Bases: Enum

Enum for valid output spaces of scores.

Sentiment

Bases: Semantics, WithPrompt

This evaluates the positive sentiment of either the prompt or response.

Sentiment is currently available to use with OpenAI, HuggingFace or Cohere as the model provider.

• The OpenAI sentiment feedback function prompts a Chat Completion model to rate the sentiment from 0 to 10, and then scales the response down to 0-1.
• The HuggingFace sentiment feedback function returns a raw score from 0 to 1.
• The Cohere sentiment feedback function uses the classification endpoint and a small set of examples stored in feedback_prompts.py to return either a 0 or a 1.

Harmfulness

Bases: Moderation, WithPrompt

Examples of Harmfulness:

Insensitivity

Bases: Semantics, WithPrompt

Examples and categorization of racial insensitivity: https://sph.umn.edu/site/docs/hewg/microaggressions.pdf .

Maliciousness

Bases: Moderation, WithPrompt

Examples of maliciousness:

Hate

Bases: Moderation

Examples of (not) Hate metrics:

• openai package: openai.moderation category hate.

HateThreatening

Bases: Hate

Examples of (not) Threatening Hate metrics:

• openai package: openai.moderation category hate/threatening.

SelfHarm

Bases: Moderation

Examples of (not) Self Harm metrics:

• openai package: openai.moderation category self-harm.

Sexual

Bases: Moderation

Examples of (not) Sexual metrics:

• openai package: openai.moderation category sexual.

SexualMinors

Bases: Sexual

Examples of (not) Sexual Minors metrics:

• openai package: openai.moderation category sexual/minors.

Violence

Bases: Moderation

Examples of (not) Violence metrics:

• openai package: openai.moderation category violence.

GraphicViolence

Bases: Violence

Examples of (not) Graphic Violence:

• openai package: openai.moderation category violence/graphic.

FeedbackOutput

Bases: BaseModel

Feedback functions produce at least a floating score.

ClassificationModel

Bases: Model

Functions

of_prompt staticmethod

of_prompt(model: CompletionModel, prompt: str) -> None

Define a classification model from a completion model, a prompt, and optional examples.

Functions