Misc. Utilities

trulens_eval.utils.generated

Utilities for dealing with LLM-generated text.

Attributes

PATTERN_0_10 module-attribute

PATTERN_0_10: Pattern = compile('([0-9]+)(?=\\D*$)')

Regex that matches the last integer.

PATTERN_NUMBER module-attribute

PATTERN_NUMBER: Pattern = compile('([+-]?[0-9]+\\.[0-9]*|[1-9][0-9]*|0)')

Regex that matches floating point and integer numbers.

PATTERN_INTEGER module-attribute

PATTERN_INTEGER: Pattern = compile('([+-]?[1-9][0-9]*|0)')

Regex that matches integers.

Classes

ParseError

Bases: Exception

Error parsing LLM-generated text.

Functions

validate_rating

validate_rating(rating) -> int

Validate a rating is between 0 and 10.

re_0_10_rating

re_0_10_rating(s: str) -> int

Extract a 0-10 rating from a string.

If the string does not match an integer or matches an integer outside the 0-10 range, raises an error instead. If multiple numbers are found within the expected 0-10 range, the smallest is returned.

PARAMETER	DESCRIPTION
s	String to extract rating from. TYPE: str

RETURNS	DESCRIPTION
int	Extracted rating. TYPE: int

RAISES	DESCRIPTION
ParseError	If no integers between 0 and 10 are found in the string.

trulens_eval.utils.pace

Classes

Pace

Bases: BaseModel

Keep a given pace.

Calls to Pace.mark may block until the pace of its returns is kept to a constraint: the number of returns in the given period of time cannot exceed marks_per_second * seconds_per_period. This means the average number of returns in that period is bounded above exactly by marks_per_second.

Attributes

marks_per_second class-attribute instance-attribute

marks_per_second: float = 1.0

The pace in number of mark returns per second.

seconds_per_period class-attribute instance-attribute

seconds_per_period: float = 60.0

Evaluate pace as overage over this period.

Assumes that prior to construction of this Pace instance, the period did not have any marks called. The longer this period is, the bigger burst of marks will be allowed initially and after long periods of no marks.

seconds_per_period_timedelta class-attribute instance-attribute

seconds_per_period_timedelta: timedelta = Field(default_factory=lambda: timedelta(seconds=60.0))

The above period as a timedelta.

mark_expirations class-attribute instance-attribute

mark_expirations: Deque[datetime] = Field(default_factory=deque)

Keep track of returns that happened in the last period seconds.

Store the datetime at which they expire (they become longer than period seconds old).

max_marks instance-attribute

max_marks: int

The maximum number of marks to keep track in the above deque.

It is set to (seconds_per_period * returns_per_second) so that the average returns per second over period is no more than exactly returns_per_second.

last_mark class-attribute instance-attribute

last_mark: datetime = Field(default_factory=now)

Time of the last mark return.

lock class-attribute instance-attribute

lock: LockType = Field(default_factory=Lock)

Thread Lock to ensure mark method details run only one at a time.

Functions

mark

mark() -> float

Return in appropriate pace. Blocks until return can happen in the appropriate pace. Returns time in seconds since last mark returned.