โญ Core Concepts¶
-
โ Feedback Functions.
-
โ Rag Triad.
Glossary¶
General and ๐ฆTruLens-Eval-specific concepts.
-
Agent
. AComponent
of anApplication
or the entirety of an application that providers a natural language interface to some set of capabilities typically incorporatingTools
to invoke or query local or remote services, while maintaining its state viaMemory
. The user of an agent may be a human, a tool, or another agent. See alsoMulti Agent System
. -
Application
orApp
. An "application" that is tracked by ๐ฆTruLens-Eval. Abstract definition of this tracking corresponds to App. We offer special support for LangChain via TruChain, LlamaIndex via TruLlama, and NeMo Guardrails via TruRailsApplications
as well as custom apps via TruBasicApp or TruCustomApp, and apps that already come withTrace
s via TruVirtual. -
Chain
. A LangChainApp
. -
Chain of Thought
. The use of anAgent
to deconstruct its tasks and to structure, analyze, and refine itsCompletions
. -
Completion
,Generation
. The process or result of LLM responding to somePrompt
. -
Component
. Part of anApplication
giving it some capability. Typical components include: -
Retriever
-
Memory
-
Tool
-
Prompt Template
-
LLM
-
Embedding
. A real vector representation of some piece of text. Can be used to find related pieces of text in aRetrieval
. -
Eval
,Evals
,Evaluation
. Process or result of method that scores the outputs or aspects of aTrace
. In ๐ฆTruLens-Eval, our scores are real numbers between 0 and 1. -
Feedback
. SeeEvaluation
. -
Feedback Function
. A method that implements anEvaluation
. This corresponds to Feedback. -
Generation
. SeeCompletion
. -
Human Feedback
. A feedback that is provided by a human, e.g. a thumbs up/down in response to aCompletion
. -
Instruction Prompt
,System Prompt
. A part of aPrompt
given to anLLM
to complete that contains instructions describing the task that theCompletion
should solve. Sometimes such prompts include examples of correct or desirable completions (seeShots
). A prompt that does not include examples is said to beZero Shot
. -
LLM
,Large Language Model
. TheComponent
of anApplication
that performsCompletion
. -
Memory
. The state maintained by anApplication
or anAgent
indicating anything relevant to continuing, refining, or guiding it towards its goals.Memory
is provided asContext
inPrompts
and is updated when new relevant context is processed, be it a user prompt or the results of the invocation of someTool
. AsMemory
is included inPrompts
, it can be a natural language description of the state of the app/agent. To limit to size if memory,Summarization
is often used. -
Multi-Agent System
. The use of multipleAgents
incentivized to interact with each other to implement some capability. While the term predatesLLMs
, the convenience of the common natural language interface makes the approach much easier to implement. -
Prompt
. The text that anLLM
completes duringCompletion
. In chat applications. See alsoInstruction Prompt
,Prompt Template
. -
Prompt Template
. A piece of text with placeholders to be filled in in order to build aPrompt
for a given task. APrompt Template
will typically include theInstruction Prompt
with placeholders for things likeContext
,Memory
, orApplication
configuration parameters. -
Provider
. A system that provides the ability to execute models, eitherLLM
s or classification models. In ๐ฆTruLens-Eval,Feedback Functions
make use ofProviders
to invoke models forEvaluation
. -
RAG
,Retrieval Augmented Generation
. A common organization ofApplications
that combine aRetrieval
with anLLM
to produceCompletions
that incorporate information that anLLM
alone may not be aware of. -
RAG Triad
(๐ฆTruLens-Eval-specific concept). A combination of threeFeedback Functions
meant toEvaluate
Retrieval
steps inApplications
. -
Record
. A "record" of the execution of a single execution of an app. Single execution means invocation of some top-level app method. Corresponds to RecordNote
This will be renamed to
Trace
in the future. -
Retrieval
,Retriever
. The process or result (or theComponent
that performs this) of looking up pieces of text relevant to aPrompt
to provide asContext
to anLLM
. Typically this is done using anEmbedding
representations. -
Selector
(๐ฆTruLens-Eval-specific concept). A specification of the source of data from aTrace
to use as inputs to aFeedback Function
. This corresponds to Lens and utilities Select. -
Shot
,Zero Shot
,Few Shot
,<Quantity>-Shot
. The use of zero or more examples in anInstruction Prompt
to help anLLM
generate desirableCompletions
.Zero Shot
describes prompts that do not have any examples and only offer a natural language description of the task, while<Quantity>-Shot
indicate some<Quantity>
of examples are provided. -
Span
. Some unit of work logged as part of a record. Corresponds to current ๐ฆRecordAppCallMethod. -
Summarization
. The task of condensing some natural language text into a smaller bit of natural language text that preserves the most important parts of the text. This can be targetted towards humans or otherwise. It can also be used to maintain consizeMemory
in anLLM
Application
orAgent
. Summarization can be performed by anLLM
using a specificInstruction Prompt
. -
Tool
. A piece of functionality that can be invoked by anApplication
orAgent
. This commonly includes interfaces to services such as search (generic search via google or more specific like IMDB for movies). Tools may also perform actions such as submitting comments to github issues. ATool
may also encapsulate an interface to anAgent
for use as a component in a largerApplication
. -
Trace
. SeeRecord
.