Skip to content

🀝 Contributing to TruLens

Interested in contributing to TruLens? Here's how to get started!

What can you work on?

  1. πŸ’ͺ Add new feedback functions
  2. 🀝 Add new feedback function providers.
  3. πŸ› Fix bugs
  4. πŸŽ‰ Add usage examples
  5. πŸ§ͺ Add experimental features
  6. πŸ“„ Improve code quality & documentation
  7. β›… Address open issues.

Also, join the AI Quality Slack community for ideas and discussions.

πŸ’ͺ Add new feedback functions

Feedback functions are the backbone of TruLens, and evaluating unique LLM apps may require new evaluations. We'd love your contribution to extend the feedback functions library so others can benefit!

  • To add a feedback function for an existing model provider, you can add it to an existing provider module. You can read more about the structure of a feedback function in this guide.
  • New methods can either take a single text (str) as a parameter or two different texts (str), such as prompt and retrieved context. It should return a float, or a dict of multiple floats. Each output value should be a float on the scale of 0 (worst) to 1 (best).
  • Make sure to add its definition to this list.

🀝 Add new feedback function providers.

Feedback functions often rely on a model provider, such as OpenAI or HuggingFace. If you need a new model provider to utilize feedback functions for your use case, we'd love if you added a new provider class, e.g. Ollama.

You can do so by creating a new provider module in this folder.

Alternatively, we also appreciate if you open a GitHub Issue if there's a model provider you need!

πŸ› Fix Bugs

Most bugs are reported and tracked in the Github Issues Page. We try our best in triaging and tagging these issues:

Issues tagged as bug are confirmed bugs. New contributors may want to start with issues tagged with good first issue. Please feel free to open an issue and/or assign an issue to yourself.

πŸŽ‰ Add Usage Examples

If you have applied TruLens to track and evalaute a unique use-case, we would love your contribution in the form of an example notebook: e.g. Evaluating Pinecone Configuration Choices on Downstream App Performance

All example notebooks are expected to:

  • Start with a title and description of the example
  • Include a commented out list of dependencies and their versions, e.g. # ! pip install trulens==0.10.0 langchain==0.0.268
  • Include a linked button to a Google colab version of the notebook
  • Add any additional requirements

πŸ§ͺ Add Experimental Features

If you have a crazy idea, make a PR for it! Whether if it's the latest research, or what you thought of in the shower, we'd love to see creative ways to improve TruLens.

πŸ“„ Improve Code Quality & Documentation

We would love your help in making the project cleaner, more robust, and more understandable. If you find something confusing, it most likely is for other people as well. Help us be better!

Big parts of the code base currently do not follow the code standards outlined in Standards index. Many good contributions can be made in adapting us to the standards.

β›… Address Open Issues

See 🍼 good first issue or πŸ§™ all open issues.

πŸ‘€ Things to be Aware Of

🧭 Design Goals and Principles

The design of the API is governed by the principles outlined in the Design doc.

βœ… Standards

We try to respect various code, testing, and documentation standards outlined in the Standards index.

πŸ’£ Tech Debt

Parts of the code are nuanced in ways should be avoided by new contributors. Discussions of these points are welcome to help the project rid itself of these problematic designs. See Tech debt index.

β›… Optional Packages

Limit the packages installed by default when installing TruLens-Eval. For optional functionality, additional packages can be requested for the user to install and their usage is aided by an optional imports scheme. See Optional Packages for details.

✨ Database Migration

Database migration.

πŸ‘‹πŸ‘‹πŸ»πŸ‘‹πŸΌπŸ‘‹πŸ½πŸ‘‹πŸΎπŸ‘‹πŸΏ Contributors

TruLens Eval Contributors

See contributors on github.

TruLens Explain Contributors (alphabetical)

🧰 Maintainers

The current maintainers of TruLens-Eval are:

Name Employer Github Name
Aaron Varghese Truera arn-tru
Corey Hu Truera coreyhu
Daniel Huang Truera daniel-huang-1230
Garett Tok Ern Liang Truera walnutdust
Josh Reini Truera joshreini1
Piotr Mardziel Truera piotrm0
Ricardo Aravena Truera raravena80
Shayak Sen Truera shayaks