📓 LlamaIndex Quickstart¶
In this quickstart you will create a simple Llama Index app and learn how to log it and get feedback on an LLM response.
You'll also learn how to use feedbacks for guardrails, via filtering retrieved context.
For evaluation, we will leverage the RAG triad of groundedness, context relevance and answer relevance.
# pip install trulens_eval llama_index openai
Add API keys¶
For this quickstart, you will need an Open AI key. The OpenAI key is used for embeddings, completion and evaluation.
import os
os.environ["OPENAI_API_KEY"] = "sk-..."
Import from TruLens¶
from trulens_eval import Tru
tru = Tru()
tru.reset_database()
🦑 Tru initialized with db url sqlite:///default.sqlite . 🛑 Secret keys may be written to the database. See the `database_redact_keys` option of `Tru` to prevent this.
Download data¶
This example uses the text of Paul Graham’s essay, “What I Worked On”, and is the canonical llama-index example.
The easiest way to get it is to download it via this link and save it in a folder called data. You can do so with the following command:
import os
import urllib.request
url = "https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt"
file_path = 'data/paul_graham_essay.txt'
if not os.path.exists('data'):
os.makedirs('data')
if not os.path.exists(file_path):
urllib.request.urlretrieve(url, file_path)
Create Simple LLM Application¶
This example uses LlamaIndex which internally uses an OpenAI LLM.
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
Settings.chunk_size = 128
Settings.chunk_overlap = 16
Settings.llm = OpenAI()
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=3)
Send your first request¶
response = query_engine.query("What did the author do growing up?")
print(response)
The author worked on writing and programming outside of school before college.
Initialize Feedback Function(s)¶
from trulens_eval.feedback.provider import OpenAI
from trulens_eval import Feedback
import numpy as np
# Initialize provider class
provider = OpenAI()
# select context to be used in feedback. the location of context is app specific.
from trulens_eval.app import App
context = App.select_context(query_engine)
# Define a groundedness feedback function
f_groundedness = (
Feedback(provider.groundedness_measure_with_cot_reasons, name = "Groundedness")
.on(context.collect()) # collect context chunks into a list
.on_output()
)
# Question/answer relevance between overall question and answer.
f_answer_relevance = (
Feedback(provider.relevance_with_cot_reasons, name = "Answer Relevance")
.on_input_output()
)
# Question/statement relevance between question and each context chunk.
f_context_relevance = (
Feedback(provider.context_relevance_with_cot_reasons, name = "Context Relevance")
.on_input()
.on(context)
.aggregate(np.mean)
)
✅ In Groundedness, input source will be set to __record__.app.query.rets.source_nodes[:].node.text.collect() . ✅ In Groundedness, input statement will be set to __record__.main_output or `Select.RecordOutput` . ✅ In Answer Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` . ✅ In Answer Relevance, input response will be set to __record__.main_output or `Select.RecordOutput` . ✅ In Context Relevance, input question will be set to __record__.main_input or `Select.RecordInput` . ✅ In Context Relevance, input context will be set to __record__.app.query.rets.source_nodes[:].node.text .
Instrument app for logging with TruLens¶
from trulens_eval import TruLlama
tru_query_engine_recorder = TruLlama(query_engine,
app_id='LlamaIndex_App1',
feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance])
# or as context manager
with tru_query_engine_recorder as recording:
query_engine.query("What did the author do growing up?")
Use guardrails¶
In addition to making informed iteration, we can also directly use feedback results as guardrails at inference time. In particular, here we show how to use the context relevance score as a guardrail to filter out irrelevant context before it gets passed to the LLM. This both reduces hallucination and improves efficiency.
Below, you can see the TruLens feedback display of each context relevance chunk retrieved by our RAG.
last_record = recording.records[-1]
from trulens_eval.utils.display import get_feedback_result
get_feedback_result(last_record, "Context Relevance")
Wouldn't it be great if we could automatically filter out context chunks with relevance scores below 0.5?
We can do so with the TruLens guardrail, WithFeedbackFilterNodes. All we have to do is use the method of_query_engine
to create a new filtered retriever, passing in the original retriever along with the feedback function and threshold we want to use.
from trulens_eval.guardrails.llama import WithFeedbackFilterNodes
# note: feedback function used for guardrail must only return a score, not also reasons
f_context_relevance_score = Feedback(provider.context_relevance)
filtered_query_engine = WithFeedbackFilterNodes(query_engine, feedback=f_context_relevance_score, threshold=0.5)
Then we can operate as normal
tru_recorder = TruLlama(filtered_query_engine,
app_id='LlamaIndex_App1_Filtered',
feedbacks=[f_answer_relevance, f_context_relevance, f_groundedness])
with tru_recorder as recording:
llm_response = filtered_query_engine.query("What did the author do growing up?")
display(llm_response)
Response(response='The author focused on writing and programming outside of school before college. Specifically, the author wrote short stories, which were described as having characters with strong feelings but lacking in plot.', source_nodes=[NodeWithScore(node=TextNode(id_='a98829e7-c59e-4906-9ec8-d1a84ab231e4', embedding=None, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='01d1924b-a1ae-4a1b-a728-02c0d2076cdd', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, hash='1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='ac44aacc-cb1f-48db-9581-1ccbc3305a4e', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='e99647d69b84b0a13c59268b58406bb00652b41196a27d7b46b56e5d713018ed')}, text="What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.", mimetype='text/plain', start_char_idx=2, end_char_idx=373, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.8208121222994761)], metadata={'a98829e7-c59e-4906-9ec8-d1a84ab231e4': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}})
See the power of context filters!¶
If we inspect the context relevance of our retreival now, you see only relevant context chunks!
last_record = recording.records[-1]
from trulens_eval.utils.display import get_feedback_result
get_feedback_result(last_record, "Context Relevance")
tru.get_leaderboard()
Groundedness | Context Relevance | Answer Relevance | latency | total_cost | |
---|---|---|---|---|---|
app_id | |||||
LlamaIndex_App1_Filtered | 1.0 | 0.8 | 0.8 | 1.0 | 0.005268 |
LlamaIndex_App1 | 0.8 | 0.4 | 0.8 | 1.0 | 0.000713 |
Retrieve records and feedback¶
# The record of the app invocation can be retrieved from the `recording`:
rec = recording.get() # use .get if only one record
# recs = recording.records # use .records if multiple
display(rec)
Record(record_id='record_hash_9f960b879e7fbb4c48a58b1cdfb87b3f', app_id='LlamaIndex_App1_Filtered', cost=Cost(n_requests=5, n_successful_requests=15, n_classes=0, n_tokens=3537, n_stream_chunks=0, n_prompt_tokens=3493, n_completion_tokens=44, cost=0.005267500000000001), perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 20, 778856), end_time=datetime.datetime(2024, 7, 3, 5, 47, 22, 824169)), ts=datetime.datetime(2024, 7, 3, 5, 47, 22, 824425), tags='-', meta=None, main_input='What did the author do growing up?', main_output='The author focused on writing and programming outside of school before college. Specifically, the author wrote short stories, which were described as having characters with strong feelings but lacking in plot.', main_error=None, calls=[RecordAppCall(call_id='74117bff-ce0f-4fd8-9f34-d299e7a8d80f', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.query_engine, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=14295036320, init_bindings=None), name='retrieve')), RecordAppCallMethod(path=Lens().app.query_engine._retriever, method=Method(obj=Obj(cls=llama_index.core.indices.vector_store.retrievers.retriever.VectorIndexRetriever, id=4348938384, init_bindings=None), name='retrieve')), RecordAppCallMethod(path=Lens().app.query_engine._retriever, method=Method(obj=Obj(cls=llama_index.core.indices.vector_store.retrievers.retriever.VectorIndexRetriever, id=4348938384, init_bindings=None), name='_retrieve'))], args={'query_bundle': {'query_str': 'What did the author do growing up?', 'image_path': None, 'custom_embedding_strs': None, 'embedding': [0.012144206091761589, -0.015698930248618126, 0.007664461154490709, -0.010062908753752708, -0.021275486797094345, 0.02139441855251789, -0.006689885165542364, -0.01778683438897133, -0.027882780879735947, -0.03047283925116062, 0.024301627650856972, 0.0009704462718218565, 0.0025801481679081917, 0.01097471546381712, 0.003690173616632819, 0.01857971027493477, 0.03898303583264351, -0.009613612666726112, -0.0009795313235372305, -0.019055435433983803, -0.0023604556918144226, 0.010723638348281384, 0.0035844568628817797, -0.008569660596549511, -0.003151679178699851, 0.006812119856476784, 0.025041643530130386, -0.028543509542942047, 0.023363390937447548, -0.0033779789227992296, 0.011450440622866154, -0.013901746831834316, -0.008450728841125965, -0.005563341546803713, -0.024883069097995758, -0.009395572356879711, -0.016082152724266052, -0.014469973742961884, 0.0017724066274240613, 0.00755874440073967, 0.016095368191599846, -0.0030955171678215265, -0.013888532295823097, -0.0018087467178702354, 0.02201550267636776, 0.007327489089220762, -0.010360237210988998, -0.0012966814683750272, -0.016068939119577408, -0.0013982686214148998, 0.03425221145153046, 0.013406199403107166, -0.004350902978330851, -0.024103408679366112, -0.019491517916321754, 0.0012314344057813287, 0.0026974277570843697, 0.014099964872002602, 0.0016691676573827863, -0.021275486797094345, -0.007023553363978863, 0.022174078971147537, -0.00299640791490674, 0.007413383573293686, -0.024750923737883568, 0.0007007861277088523, -0.013194765895605087, -0.005199940409511328, -0.008424299769103527, -0.0153157077729702, 0.035996537655591965, 0.00011222076136618853, -0.009904333390295506, -0.0027948853094130754, 0.019002577289938927, -0.010941678658127785, -0.00964004173874855, -0.007367132697254419, -0.009957191534340382, 0.010611314326524734, -0.00785607285797596, -0.0219230018556118, -0.0064322007820010185, 0.009560754522681236, 0.01099453680217266, -0.005642629228532314, 0.016808954998850822, 0.029627105221152306, -0.020257962867617607, -0.01844756491482258, -0.0026214439421892166, 0.017601830884814262, 0.01657109335064888, 0.015130703337490559, 0.014773909002542496, 0.0032887805718928576, -0.01564607210457325, 0.01129186525940895, -0.008840559050440788, -0.003425881965085864, -0.005804507527500391, 0.0041592917405068874, -0.007195343263447285, -0.012401890940964222, -0.01877792924642563, 0.013036190532147884, -0.0062967510893940926, -0.021909786388278008, 0.028305647894740105, -0.03242859989404678, -0.014086750335991383, 0.0202711783349514, 0.02759205922484398, -0.05277906358242035, -0.008543231524527073, -0.025226648896932602, 0.0007338225841522217, 0.000338417332386598, 0.008952883072197437, 0.0009630130953155458, 0.007968396879732609, 0.001654301187954843, 0.021962644532322884, -0.010399880819022655, 0.00944182276725769, -0.0012454749085009098, -0.015897149220108986, -0.00881412997841835, -0.011126683093607426, -0.007102841045707464, 0.017469685524702072, -0.0016848599771037698, 0.012481178157031536, -0.013478879816830158, -0.017667904496192932, 0.016610736027359962, -0.019029006361961365, 0.015342136844992638, -0.021473705768585205, -0.02525307796895504, 0.020482610911130905, 0.022914094850420952, -0.008179829455912113, 0.002928683068603277, 0.006983909755945206, 0.018473993986845016, 0.030552126467227936, 0.039062321186065674, 0.01008273009210825, -0.018698642030358315, 0.017469685524702072, -0.0025884073693305254, 0.007089626509696245, -0.008457336574792862, 0.01437747199088335, 0.035071514546871185, -0.015381780453026295, 0.012540644034743309, -0.016002865508198738, -0.01994081400334835, 0.0209319069981575, 0.025689158588647842, 0.0221476498991251, -0.008212866261601448, -0.01170151773840189, 0.03816372901201248, 0.023416249081492424, 0.014958913438022137, 0.0028097517788410187, -0.012679397128522396, -0.024050550535321236, 0.012593502178788185, -0.03351219370961189, 0.009435215964913368, 0.007281237747520208, 0.024235554039478302, -0.0026924721896648407, 0.015540354885160923, -0.01706003211438656, -0.003908214159309864, -0.012666182592511177, 0.001992099219933152, 0.02501521445810795, 0.04424244165420532, -0.008622518740594387, -0.00881412997841835, 0.00861591100692749, 0.008080720901489258, -0.004327777307480574, 0.0031896710861474276, -0.012124384753406048, 0.024473417550325394, -0.012501000426709652, -0.02449984662234783, -0.66982102394104, -0.012752077542245388, -0.0032127967569977045, -0.01967652142047882, 0.01149008423089981, 0.022755520418286324, -0.002528941724449396, 0.0365779809653759, -0.0306314155459404, 0.004301348235458136, -0.007320881821215153, 0.03858659788966179, 0.026719896122813225, 0.0004080004000570625, 0.011866699904203415, -0.011476869694888592, 0.010109160095453262, -0.02218729257583618, 0.00853001605719328, -0.015342136844992638, 0.00508431252092123, 0.02073368802666664, -0.007624817080795765, -0.0062571074813604355, 0.008576267398893833, 0.010743459686636925, 0.008992526680231094, -0.03311575576663017, 0.013835673220455647, 0.02571558766067028, -0.017839694395661354, 0.028807802125811577, 0.03647226095199585, 0.025794874876737595, 0.05513126030564308, -0.012322602793574333, -0.008378048427402973, 0.03536223620176315, 0.005566644947975874, 0.04387243092060089, -0.02062797173857689, -0.016676809638738632, 0.015619643032550812, 0.014205682091414928, 0.007710712030529976, -0.01119275577366352, 0.039564475417137146, 0.0032887805718928576, 0.035071514546871185, 0.019266869872808456, -0.005854062270373106, -0.001008438179269433, -0.001815353985875845, -0.007922145538032055, -0.0007602517725899816, -0.006504880730062723, 0.0012182198697701097, 0.015421424061059952, 0.007294452283531427, 0.017800049856305122, -0.006389253307133913, 0.03715942054986954, -0.023033026605844498, 0.004704393446445465, -0.0033829344902187586, 0.02241194061934948, -0.016544664278626442, 0.026204528287053108, -0.005606289021670818, -0.022636588662862778, 0.009778794832527637, 0.007585173472762108, -0.015289277769625187, -0.009474859572947025, 0.01335334125906229, 0.010538633912801743, 0.03274574875831604, 0.005295746028423309, -0.002448002342134714, -0.006716314237564802, -0.027023831382393837, -0.0037000845186412334, -0.04067450016736984, -0.010849176906049252, 0.02205514721572399, 0.006448718719184399, -0.03023497760295868, 0.007922145538032055, -0.003838837845250964, 0.007677675690501928, -0.01026773452758789, 0.03277217969298363, -0.007690890226513147, -0.026019522920250893, -0.012540644034743309, 0.009811831638216972, -0.013445843011140823, -0.001188487047329545, 0.0007317577837966383, 0.002581800101324916, 0.010611314326524734, -0.002264650072902441, 0.010221484117209911, 0.011338116601109505, 0.00550387566909194, 0.000641320482827723, 0.012434926815330982, 0.025530584156513214, 0.03927375376224518, -0.018923290073871613, 0.005708701908588409, 0.0009473207755945623, 0.0013354993425309658, -0.002353848423808813, -0.007869287393987179, -0.023151958361268044, 0.014033892191946507, 0.008292153477668762, 0.011001144535839558, -0.013459057547152042, -0.0014023981057107449, 0.010413095355033875, -0.00911145843565464, -0.018209701403975487, -0.005014935974031687, 0.023455893620848656, -0.009197353385388851, 0.010703816078603268, -0.011357937939465046, 0.0007561221718788147, -0.0007342355675064027, -0.010023265145719051, 0.013300483115017414, -0.01910829357802868, 0.013769600540399551, 0.0031285537406802177, -0.003805801272392273, -0.0061943382024765015, 0.00477707339450717, -0.00725480867549777, -0.020509039983153343, 0.004998418036848307, -0.005209851078689098, -0.003214448457583785, -0.017152534797787666, -0.02046939730644226, 0.01117293443530798, -0.002784974407404661, -0.012765292078256607, 0.02152656391263008, 0.003366416320204735, -0.013518523424863815, -0.022517656907439232, -0.015883933752775192, -0.01828899048268795, -0.016201084479689598, 0.01265957485884428, -0.005351908039301634, -0.01431139837950468, 0.009085029363632202, -0.0023109009489417076, 0.03628725931048393, -0.010928464122116566, 0.010155410505831242, -0.014218896627426147, 0.01036684401333332, 0.013181551359593868, 0.012296173721551895, 0.010868998244404793, -0.008701805956661701, 0.011272042989730835, -0.016148226335644722, 0.0034490074031054974, -0.005170207470655441, -0.007446420378983021, 0.012652968056499958, 0.0008837255882099271, -0.002722205128520727, -0.008477157913148403, 0.004420279525220394, -0.004479745402932167, -0.007902323268353939, -0.007882501929998398, -0.011575979180634022, 0.023706970736384392, -0.009382357820868492, 0.0012628190452232957, 0.03245502710342407, 0.016201084479689598, 0.03227002173662186, 7.371262472588569e-05, 0.014536046423017979, 0.006762565113604069, 0.009058600291609764, -0.0006429722998291254, -0.009461645036935806, 0.030049972236156464, 0.007049982436001301, 0.009930762462317944, 0.00785607285797596, -0.026191312819719315, 0.026191312819719315, 0.01399424858391285, 0.0019755808170884848, 0.017245037481188774, -0.037317994982004166, -0.004004020243883133, -0.02023153379559517, -0.008219473995268345, 0.0025223344564437866, 0.003528294852003455, -0.008153400383889675, 0.006025852169841528, -0.018989363685250282, -0.01437747199088335, 0.006141479592770338, -0.018077556043863297, 0.017152534797787666, -0.01047916803508997, 0.001551062217913568, 0.017932195216417313, -0.011873307637870312, -0.004704393446445465, -0.001744325621984899, -0.015223205089569092, 0.019861524924635887, 0.001973929116502404, -0.019385799765586853, 0.006937658414244652, -0.022068360820412636, -0.004836539272218943, -0.004796895198523998, 0.01090864185243845, 0.01802469789981842, -0.005242887884378433, 0.021143341436982155, 0.02218729257583618, -0.01544785313308239, 0.01835506223142147, -0.011675088666379452, -0.00881412997841835, 0.016861815005540848, 0.024737708270549774, -0.005655843764543533, 0.030552126467227936, -0.011959201656281948, 0.017773620784282684, -0.021169770509004593, -0.011893128976225853, -0.0057483455166220665, 0.003977591171860695, -0.012084740214049816, 0.0026743023190647364, -0.023680541664361954, 0.006250500213354826, 0.007895716466009617, 0.005355211906135082, -0.004192328080534935, 0.00868198461830616, 0.02957424707710743, -0.0042352755554020405, 0.009626827202737331, -0.003957768902182579, 0.01706003211438656, 0.0014511268818750978, 0.013703527860343456, -0.009659864008426666, -0.005081009119749069, -0.03219073638319969, -0.0105782775208354, -0.022755520418286324, -0.010644350200891495, 0.006716314237564802, -0.02126227132976055, 0.010743459686636925, 0.007307667285203934, -0.00924360379576683, -0.004988506902009249, 0.00911145843565464, -0.0010546892881393433, -0.02261015959084034, -0.0025702372658997774, 0.022821594029664993, 0.01716575026512146, 0.0043608141131699085, -0.007003731559962034, -0.007076411973685026, 0.0016361312009394169, -0.014033892191946507, -0.0004860490735154599, 0.005262709688395262, -0.002244828036054969, 0.0026528285816311836, -0.010849176906049252, 0.0041592917405068874, 0.016465377062559128, 0.03020854853093624, -0.023588038980960846, 0.023521967232227325, -0.02099798060953617, -0.03345933556556702, -0.009177531115710735, -0.022359082475304604, 0.0017509328899905086, 0.02838493511080742, 0.005150385666638613, -0.014641763642430305, -0.013095656409859657, -0.007003731559962034, -0.007023553363978863, -0.012302781455218792, 0.0009704462718218565, 0.0061447834596037865, 0.01338637713342905, 0.010360237210988998, -0.011278650723397732, -0.014694621786475182, -0.006039066705852747, 0.015936793759465218, 0.0032028856221586466, -0.026415960863232613, -0.021275486797094345, -0.029124950990080833, 0.012104562483727932, 0.1068795844912529, 0.014403901062905788, -0.01564607210457325, 0.015077845193445683, 0.009785402566194534, -0.011424011550843716, -0.016161441802978516, -0.021804070100188255, 0.01004969421774149, -0.0018682123627513647, -0.00121904572006315, -0.011873307637870312, 0.01411317940801382, -0.013716742396354675, -0.007076411973685026, -0.0016295238165184855, -3.1307215976994485e-05, -0.025755232200026512, -0.01523641962558031, -0.021315129473805428, 0.003528294852003455, 0.026230957359075546, 0.010095945559442043, 0.022636588662862778, 0.007809821516275406, -0.006267018150538206, 0.004080004058778286, 0.0010505596874281764, 0.0286492258310318, -0.01076328195631504, 0.00599281582981348, 0.0006190208368934691, 0.009461645036935806, -0.015513925813138485, 0.006415682379156351, 0.04109736904501915, 0.014086750335991383, 0.0287549439817667, 0.02525307796895504, -0.0040106275118887424, 0.031635724008083344, -0.009428608231246471, 0.014205682091414928, -0.0352565199136734, 0.014060321263968945, -0.026363102719187737, 0.0026974277570843697, 0.028173500671982765, 0.0040866113267838955, 0.0007334096007980406, 0.0024314841721206903, -0.0022894274443387985, -0.0057483455166220665, -0.007684282958507538, 0.014998557046055794, -0.00549726840108633, 0.013848887756466866, -0.0032161003910005093, -0.0018748196307569742, 0.02627060003578663, -0.010849176906049252, -0.031609293073415756, 0.009283248335123062, -0.026257386431097984, 0.007301060017198324, -0.02719562128186226, -0.013207980431616306, 0.0042716157622635365, -0.009501288644969463, -0.002657783916220069, -0.009236996993422508, -0.00550387566909194, -0.038110870867967606, -0.015831075608730316, 0.024024121463298798, 0.01835506223142147, 0.039564475417137146, -0.004829932004213333, -0.008424299769103527, 0.00785607285797596, -0.0031615900807082653, -0.02994425594806671, 0.018500423058867455, -0.018764715641736984, 0.00019966416584793478, 0.024830210953950882, -0.0059201354160904884, -0.01450961735099554, -0.020614756271243095, 0.008741449564695358, 0.0012396934907883406, 0.00745963491499424, -0.004582158289849758, -0.04244525730609894, -0.012798327952623367, -0.023218030110001564, -0.010670779272913933, 0.01257367990911007, 0.01641251891851425, -0.014443544670939445, 0.0061943382024765015, -0.023257674649357796, -0.028702083975076675, -0.006554435472935438, -0.0042484900914132595, -0.0007796607096679509, -0.005758256651461124, 0.019293298944830894, -0.022914094850420952, -0.02851708047091961, 0.03766157478094101, -0.006554435472935438, 0.0002335265453439206, 0.017588617280125618, 0.015791432932019234, 0.0345957912504673, -0.0055104829370975494, 0.008113756775856018, 0.0038586596492677927, -0.011324902065098286, -0.002576844533905387, -0.02644238993525505, 0.020680829882621765, 0.018460778519511223, -0.037873007357120514, -0.009085029363632202, -0.008060898631811142, -0.02225336618721485, -0.018870431929826736, -0.005137171130627394, 0.0004455793823581189, 0.02745991386473179, -0.031239286065101624, -0.013769600540399551, -0.012157420627772808, 0.00023311359109357, -0.007935360074043274, -5.884621259610867e-06, 0.002801492577418685, 0.0002467411395628005, -0.023482322692871094, -0.01690145768225193, -0.005612896289676428, -0.018619354814291, 0.002233265433460474, -0.02706347592175007, -0.003321816911920905, -0.006931051146239042, -0.012712433934211731, 0.03874517232179642, -0.008477157913148403, 0.0021110305096954107, 0.010003442876040936, 0.017918981611728668, -0.026521677151322365, -0.02571558766067028, -0.008721628226339817, 0.020125817507505417, 0.04067450016736984, -0.0031467238441109657, 0.019332941621541977, -0.00912467297166586, 0.03462221845984459, 0.008259117603302002, -0.00028266830486245453, 0.01584429107606411, 0.0076578538864851, 0.0031037763692438602, -0.005315567832440138, 0.03895660489797592, 0.014932484365999699, 0.0211565550416708, -0.0019425443606451154, 0.0017311109695583582, 0.007135877385735512, 0.024843424558639526, 0.0022084880620241165, -0.004228668287396431, -9.100515308091417e-05, -0.03557366877794266, 0.005692183505743742, 0.014218896627426147, -0.007109448313713074, 0.011252221651375294, 0.008622518740594387, 0.002487646182999015, 0.03425221145153046, 0.005391551647335291, 0.010545240715146065, -0.02267623320221901, 0.02231943979859352, -0.015302492305636406, 0.03382934629917145, -0.018170058727264404, -0.005325478967279196, -0.014575690031051636, -0.017311109229922295, -0.02119619958102703, 0.012421712279319763, 0.02161906659603119, 0.007836250588297844, 0.008358227089047432, -0.00020957511151209474, -0.013558167032897472, -0.00010520051000639796, -0.016082152724266052, 0.008371441625058651, 9.972884436137974e-05, 0.015209990553557873, 0.005536912474781275, 0.018196487799286842, -0.04149380698800087, -0.007505885791033506, -0.01712610572576523, 0.0013462360948324203, -0.01857971027493477, -0.018170058727264404, 0.023786257952451706, 0.0035745459608733654, -0.00955414678901434, -0.022914094850420952, -0.010750067420303822, 0.030974993482232094, -0.005814418662339449, 0.002353848423808813, -0.01875150017440319, -0.02194943092763424, -0.005705398507416248, 0.010717030614614487, 0.002978237811475992, -0.01326744630932808, 0.012692611664533615, 0.011794019490480423, 0.01437747199088335, -0.006904622074216604, 0.005127259995788336, 0.018341848626732826, -0.026455605402588844, -0.01613501086831093, -0.00467796390876174, 0.006508184596896172, -0.025966664776206017, 0.012395283207297325, -0.008219473995268345, -0.021605851128697395, 0.02666703797876835, -0.005768167786300182, -0.008543231524527073, -0.008800915442407131, -0.017945410683751106, -0.016068939119577408, 0.0101355891674757, 0.01584429107606411, 0.00632978742942214, -0.01809077151119709, 0.01089542731642723, -0.005470839329063892, -0.029257098212838173, 0.015355351381003857, 0.014258540235459805, -0.013571381568908691, 0.0068319421261549, 0.0018533458933234215, 0.02013903111219406, 0.0037661574315279722, -0.0012809891486540437, 0.0025355489924550056, -0.017733976244926453, -0.024209124967455864, 0.026402747258543968, 0.00548075046390295, -0.005021543242037296, -0.02036367915570736, 0.01242832001298666, 0.0077701774425804615, 0.026521677151322365, -0.018923290073871613, -0.009884512051939964, 0.016690025106072426, -0.009303069673478603, 0.010888820514082909, 0.004321170039474964, -0.03562653064727783, 0.009098243899643421, -0.009382357820868492, -0.018302204087376595, -0.014892840757966042, -0.011457047425210476, 0.012884222902357578, -0.031609293073415756, -0.04247168451547623, -0.0033251207787543535, 0.011014359071850777, 0.012553858570754528, -0.03647226095199585, -0.003501865779981017, 0.015738574787974358, 0.0273277685046196, -0.029759252443909645, 0.012606716714799404, -0.007228379603475332, -0.013412807136774063, -0.007677675690501928, 0.005880491808056831, -0.012699219398200512, -0.008444122038781643, 0.039035893976688385, -0.0013478879118338227, -0.025160575285553932, -0.009316284209489822, -0.0010183491976931691, -0.013201373629271984, -0.022570516914129257, -0.009580575861036777, -0.023521967232227325, 0.008933061733841896, -0.008767878636717796, -0.02849065139889717, -0.008193044923245907, 0.010188447311520576, 0.004208846017718315, -0.004070092923939228, 0.004790287930518389, -0.004780377261340618, -0.004453316330909729, 0.0062934476882219315, -0.003538205986842513, -0.0028543509542942047, -0.024975571781396866, 0.004783680662512779, 0.0001777775032678619, 0.014575690031051636, -0.022861236706376076, 0.00725480867549777, -0.02069404534995556, 0.002936942270025611, -0.0015444549499079585, 0.03364434093236923, 0.021975859999656677, 0.0025124235544353724, -0.003954465501010418, 0.028702083975076675, 0.005563341546803713, 0.012454749085009098, 0.01544785313308239, -0.009858082979917526, -0.0027106422930955887, 0.000394372851587832, -0.04323813319206238, -0.01038005854934454, 0.016346445307135582, 0.036392975598573685, 0.009904333390295506, -0.01277189888060093, -0.01957080513238907, -0.012296173721551895, -0.03821658715605736, -0.00755213713273406, 0.00736052542924881, 0.0006103487685322762, 0.017271466553211212, -0.008510194718837738, -0.007043375167995691, 0.007915537804365158, -0.017271466553211212, -0.0021870143245905638, -0.0005079357069917023, -0.010802925564348698, -0.020416539162397385, 0.01014880370348692, 0.01877792924642563, -0.001067077973857522, -0.0005087616154924035, -0.004076700191944838, 0.0007639683899469674, -0.011675088666379452, -0.002487646182999015, -0.02304624207317829, -0.01034702267497778, -0.002573540899902582, -0.009633434936404228, 0.03575867414474487, 0.013009761460125446, 0.00035803273203782737, 0.004268311895430088, -0.0286492258310318, -0.00962022040039301, -0.0018748196307569742, -0.020654400810599327, -0.0026511766482144594, -0.010062908753752708, 0.016623951494693756, -0.004195631481707096, 0.008675376884639263, 0.008232688531279564, -0.014932484365999699, 0.0153157077729702, 0.014284969307482243, 0.027116334065794945, -0.020773332566022873, -0.01881757378578186, 0.025134146213531494, -0.00631326949223876, -0.01242832001298666, -0.022663017734885216, 0.021711567416787148, -0.00233072298578918, -0.0020135727245360613, -0.014456759206950665, -0.03597010672092438, 0.014747479930520058, 0.00012626126408576965, -0.004995114170014858, -0.006785690784454346, 0.004674660507589579, -0.0038322305772453547, -0.0015279367798939347, 0.011113468557596207, 0.012593502178788185, -0.0002702796191442758, -0.019266869872808456, 0.006729528773576021, 0.013901746831834316, -0.006138176191598177, -0.00716230645775795, -0.0254512969404459, 0.02538522332906723, -0.024090193212032318, -0.021037623286247253, 0.005877187941223383, -0.013042798265814781, -0.0022217025980353355, -0.017522543668746948, 0.012481178157031536, -0.0006355390651151538, 0.004017234779894352, 0.24145695567131042, 0.006864978466182947, -0.0040866113267838955, 0.011364545673131943, -0.023984476923942566, 0.007188735995441675, 0.03734442591667175, 0.00950789637863636, -0.028702083975076675, 0.015672501176595688, -0.01670323871076107, -0.021605851128697395, 0.01712610572576523, 0.0035844568628817797, 0.007340703625231981, -0.011556156910955906, -0.029653534293174744, -0.0013875317526981235, -0.002456261543557048, -0.025808090344071388, 0.01828899048268795, -0.011853485368192196, -0.029098521918058395, -0.015461067669093609, 0.01877792924642563, 0.036260828375816345, 0.001434608711861074, 0.0008007214055396616, 0.010604706592857838, 0.005728523712605238, -0.0016047465614974499, -0.009871297515928745, 0.0009522762265987694, -0.011945987120270729, -0.027644917368888855, 0.0025983182713389397, 0.018738284707069397, -0.003311906009912491, -0.003343290649354458, -0.007049982436001301, 0.01822291687130928, 0.0036868699826300144, 0.011483476497232914, -0.007915537804365158, -0.021235842257738113, 0.05407409369945526, -0.01617465540766716, 0.0019029006361961365, -0.004714304115623236, 0.02056189812719822, -0.004892701283097267, -0.002264650072902441, 0.01961044780910015, 0.01904222182929516, -0.024962356314063072, 0.0018219612538814545, 0.011615622788667679, 0.007829642854630947, -0.007466242182999849, 0.01398103404790163, 0.00755213713273406, 0.02009938843548298, -0.02432805672287941, 0.01046595349907875, 0.00507440185174346, 0.012481178157031536, -0.011311687529087067, -0.004631713032722473, -0.0025933629367500544, -0.04506174474954605, -0.0015923578757792711, -0.008549838326871395, -0.021737996488809586, 0.00784285832196474, -0.026812398806214333, -0.014853197149932384, 0.004948863293975592, 0.03102785162627697, 0.018526852130889893, 0.026852043345570564, -0.0036241007037460804, -0.006055585108697414, 0.006878193002194166, -0.0007602517725899816, -0.0319528728723526, -0.04545818269252777, 0.01411317940801382, 0.0045359074138104916, -0.006713010836392641, -0.02756563015282154, 0.0008073287317529321, -0.0030294442549347878, -0.006765868980437517, 0.003452311037108302, 0.002398447599261999, -0.012071525678038597, -0.015209990553557873, 0.03385577350854874, -0.00861591100692749, -0.005451017525047064, -0.014562475495040417, -0.009177531115710735, 0.0221476498991251, 0.009375750087201595, -0.010043086484074593, 0.013505308888852596, 0.0018450868083164096, 0.01795862428843975, 0.0042848302982747555, -0.028464222326874733, -0.0039610727690160275, -0.032957181334495544, -0.0020251355599611998, 0.0030459624249488115, 0.008153400383889675, -0.0034853476099669933, -0.0031731529161334038, -0.021381203085184097, 0.006422289647161961, -0.005599681753665209, -0.03340647742152214, -0.001721200067549944, 0.002978237811475992, 0.018923290073871613, -0.003947858233004808, -0.003125250106677413, -0.019993672147393227, -0.002821314614266157, 0.005527001339942217, -0.046594638377428055, 0.01637287437915802, -0.015989651903510094, 0.003059177193790674, 0.00932289194315672, -0.012606716714799404, 0.01842113584280014, -0.010043086484074593, 0.001954107079654932, 0.0023241157177835703, 0.017324324697256088, 0.012375461868941784, -0.022068360820412636, 0.01736396923661232, 0.00045342554221861064, 0.007849465124309063, -0.03541509434580803, 0.012983332388103008, 0.02185692824423313, 0.004020538181066513, -0.029600676149129868, -0.008252509869635105, -0.004611891228705645, -0.012976725585758686, 0.01213099155575037, 0.010360237210988998, -0.04413672164082527, -0.04458601772785187, -0.027644917368888855, -0.011258828453719616, -0.009058600291609764, -0.007466242182999849, 0.012487785890698433, 0.018632568418979645, 0.010413095355033875, -0.0025173788890242577, -0.00808732770383358, -0.1692524403333664, 0.02666703797876835, 0.023429464548826218, -0.020376894623041153, 0.005873884540051222, -0.0025751928333193064, 0.033195044845342636, -0.01036684401333332, -0.01672966778278351, 0.024750923737883568, 0.019319728016853333, 0.006128265056759119, -0.02013903111219406, 0.02284802310168743, -0.007512493059039116, 0.013009761460125446, -0.014588904567062855, 0.03290432319045067, 0.019399015232920647, 0.0063694315031170845, 0.03182072937488556, 0.008536623790860176, 0.012798327952623367, 0.002411662368103862, 0.01339959166944027, 0.01505141519010067, 0.003465525573119521, 0.0018913379171863198, 0.0018748196307569742, -0.005150385666638613, 0.008734842762351036, -8.58432031236589e-05, -0.006950873415917158, -0.022623375058174133, 0.0300764013081789, 0.008820737712085247, -0.00755874440073967, -0.002223354298621416, 0.001600616960786283, 0.015897149220108986, 0.02970639429986477, 0.02089226432144642, 0.007789999712258577, 0.008093935437500477, -0.011424011550843716, 0.026230957359075546, 0.014086750335991383, -0.002994755981490016, 0.01379602961242199, -0.02013903111219406, 0.01437747199088335, -0.023099100217223167, -0.0015428031329065561, 0.00021948604262433946, 0.014879626221954823, -0.005137171130627394, 0.015817862004041672, 0.025279507040977478, -0.007611602544784546, -0.014549260959029198, 0.009435215964913368, -0.02227979525923729, 0.029098521918058395, 0.010307378135621548, 0.008503586985170841, -0.013056012801826, -0.014813552610576153, -0.015804646536707878, -0.03205858916044235, 0.021182984113693237, -0.02023153379559517, -0.008100542239844799, 0.01297011785209179, -0.0009754017810337245, 0.0031235981732606888, 0.004443405196070671, -0.004707696847617626, -0.027777064591646194, 0.0005570774665102363, 0.03594367951154709, -0.017086463049054146, 0.013320304453372955, -0.008021255023777485, 0.004902611952275038, -0.010703816078603268, 0.025279507040977478, -0.013505308888852596, 0.015461067669093609, -0.00882734451442957, -0.013379770331084728, 0.023482322692871094, -0.018936503678560257, -0.029600676149129868, -0.012421712279319763, 0.026164883747696877, 0.009910941123962402, 0.004843146540224552, 0.0033019951079040766, 0.0022117916960269213, -0.022517656907439232, 0.007796606980264187, -0.026759540662169456, 0.01180062722414732, -0.020218320190906525, 0.032296452671289444, 0.003075695363804698, 0.011298472993075848, 0.00964004173874855, 0.033353619277477264, -0.010300771333277225, -0.001560147269628942, -0.004195631481707096, 0.004839842673391104, 0.007268023211508989, -0.016544664278626442, 0.0378994382917881, -0.006105139385908842, -0.007083019241690636, 0.0019227225566282868, -0.005424588453024626, 0.058778487145900726, 0.014443544670939445, -0.027222050353884697, 0.001443693763576448, 0.0004645753651857376, 0.0009390616323798895, -0.12326567620038986, -0.004929041489958763, 0.0027453305665403605, -0.0024446987081319094, 0.01943865790963173, 0.00436411751434207, -0.0273277685046196, -0.014496402814984322, -0.009283248335123062, 0.009692899882793427, -0.009362535551190376, -0.013511915691196918, 0.0024678243789821863, -0.019782237708568573, -0.0015386735321953893, -0.010545240715146065, 0.01891007460653782, -0.02412983775138855, -0.01692788675427437, 0.040330920368433, -0.006898014806210995, -0.005543519742786884, -0.010902035050094128, -0.012633145786821842, 0.004862968344241381, -0.01894971914589405, -0.03456936031579971, 0.0010034827282652259, 0.006818727124482393, 0.005305657163262367, -0.0012760336976498365, -0.025557013228535652, 0.0017740584444254637, -0.013102264143526554, 0.0159764364361763, -0.00862912554293871, -0.01150329876691103, 0.011919558048248291, 0.012395283207297325, -0.01868542656302452, -0.0021407632157206535, 0.01168830320239067, 0.01954437606036663, -0.05092902109026909, -0.003052569692954421, -0.00014876735804136842, -0.020614756271243095, 0.017773620784282684, 0.008992526680231094, -0.014760694466531277, -0.017733976244926453, 0.016280371695756912, -0.025160575285553932, -0.013928175903856754, -0.007717319298535585, 0.0004459923366084695, 0.01024791318923235, 0.009527717716991901, 0.008444122038781643, 0.021909786388278008, -0.01577821746468544, -0.0107963178306818, -0.005705398507416248, -0.019055435433983803, 0.015394994989037514, -0.007968396879732609, -0.004935648757964373, -0.0021143341436982155, -0.014060321263968945, -0.05248834192752838, -0.030393552035093307, -0.004734125919640064, -0.028702083975076675, 0.0042484900914132595, 0.0007197820814326406, -0.000820130342617631, -0.013967819511890411, 0.0022745609749108553, 0.011714732274413109, -0.02597988024353981, -0.010175232775509357, -0.004843146540224552, 0.009732544422149658, -0.028147071599960327, 0.015791432932019234, 0.0016509975539520383, 0.0004214214568492025, 0.009646649472415447, -0.0008011343888938427, 0.00010509727144381031, -0.03205858916044235, 0.028332076966762543, -0.00036567242932505906, -0.013743171468377113, -0.007102841045707464, 0.007426598574966192, -0.001273555913940072, -0.0031103836372494698, 0.009091636165976524, -0.0016782527090981603, -0.0084837656468153, 0.0016485198866575956, -0.03895660489797592, 0.021909786388278008, -0.009712722152471542, -0.01256046537309885, 0.0153157077729702, -0.004810110200196505, 0.004344295710325241, -0.025504155084490776, -0.03523009270429611, 0.006055585108697414, -0.009877904318273067, 0.02743348479270935, 0.021235842257738113, -0.016848599538207054, -0.0143246129155159, -0.021473705768585205, 0.005804507527500391, 0.025755232200026512, 0.007003731559962034, 0.02238551154732704, 0.0013412806438282132, -0.009712722152471542, 0.026323458179831505, 0.009930762462317944, -0.010320592671632767, 0.0041758096776902676, -0.022398727014660835, 0.015751788392663002, -0.014020677655935287, -0.0041758096776902676, 0.0028411364182829857, -0.02397126331925392, 0.009085029363632202, 0.020852619782090187, 0.019399015232920647, 0.008008040487766266, 0.004093218594789505, 0.026112025603652, 0.016597522422671318, 0.06099853664636612, -0.009065207093954086, -0.01683538407087326, -0.005424588453024626, -0.03721227869391441, 0.023033026605844498, -0.010915249586105347, -0.02954781800508499, 0.019055435433983803, 0.025557013228535652, 0.004823324736207724, 0.02142084762454033, 0.011397582478821278, -0.006898014806210995, -0.007122662849724293, -0.0038751778192818165, -0.037238709628582, 0.0024446987081319094, 0.008120364509522915, -0.01150329876691103, -0.015024986118078232, 0.03179429844021797, -0.010948286391794682, 0.001663386239670217, -0.02373339980840683, 0.015461067669093609, -0.016399303451180458, 0.01044613216072321, 0.012818150222301483, -0.006871585734188557, -0.033750057220458984, -0.010571670718491077, -0.002182058757171035, 0.011542942374944687, 0.03208502009510994, 0.01110686082392931, 0.0022084880620241165, -0.001183531479910016, 0.03803158551454544, -0.005236280616372824, 0.02944210171699524, -0.009137887507677078, 0.01119275577366352, -0.0027189014945179224, 0.021777641028165817, 0.023521967232227325, 0.0235748253762722, 0.010320592671632767, -0.0016832081601023674, -0.002233265433460474, 0.003980894573032856, -0.003282173303887248, -0.01047916803508997, 0.004853057209402323, -0.01173455361276865, -0.002753589767962694, 0.02716919220983982, -0.007499278523027897, 0.008245903067290783, 0.01924044080078602, 0.01505141519010067, 0.015950007364153862, -0.01472105085849762, 0.009078421629965305, -0.015091059729456902, -0.022174078971147537, 0.04191667214035988, -0.0032408777624368668, -0.028305647894740105, 0.003954465501010418, 0.021103696897625923, 0.005384944379329681, -0.0009572317358106375, -0.010637743398547173, 0.0107963178306818, -0.015077845193445683, 0.018923290073871613, -0.0033862381242215633, -0.013307089917361736, -0.010789711028337479, 0.034410785883665085, -0.00445661973208189, 0.004189024213701487, 0.009547539986670017, -0.022028718143701553, -0.012732255272567272, 0.0244602020829916, 0.012210278771817684, 0.0038091049063950777, 0.006210856139659882, -0.018989363685250282, -0.009098243899643421, -0.01604251004755497, -0.010056301020085812, -0.008126971311867237, 0.011774198152124882, -0.011457047425210476, -0.0014808597043156624, 0.019887953996658325, -0.036234401166439056, 0.02089226432144642, -0.002771759871393442, -0.006891407538205385, -0.011014359071850777, 0.013089049607515335, 0.01961044780910015, 0.014126394875347614, -0.007968396879732609, -0.012203671969473362, -0.0037496392615139484, 0.015011771582067013, -0.0013957908377051353, 0.029785681515932083, -0.028939947485923767, -0.033591482788324356, -0.012012060731649399, 0.009018956683576107, -0.006191034335643053, 0.015157132409512997, 0.023482322692871094, 0.018315419554710388, 0.008642340078949928, 0.02327089011669159, 0.011001144535839558, -0.021011194214224815, 0.0073539181612432, 0.037978727370500565, -0.017271466553211212, -0.006696492433547974, -0.038507308810949326, -0.004694482311606407, 0.005943261086940765, -0.0559769943356514, -0.024869853630661964, 0.01431139837950468, -0.03377648815512657, -0.009805223904550076, -0.027089904993772507, -0.0004559032677207142, 0.013128693215548992, -0.012534036301076412, 0.003640618873760104, -0.02175121195614338, 0.0084837656468153, 0.004426886793226004, -0.003792586736381054, -0.004998418036848307, -0.0017971839988604188, -0.02838493511080742]}}, rets=[{'node': {'id_': 'ee3bc977-bc7e-4393-9e00-9bd7719f7c1b', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': '538dc6fd-bdfd-42da-b1c2-17ea82554a7d', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '5042ef58fa439b89369e3a10cacb784e1845e881271729a3261403a515d7134b'}, <NodeRelationship.NEXT: '3'>: {'node_id': '62d8a138-8b16-402a-a7cf-2e6dab6633f2', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '0b4f822ad12b07d089691e8fe465cfd1a8eb40b43de3358e4fe2ed3606b289e6'}}, 'text': 'I remember taking the boys to the coast on a sunny day in 2015 and figuring out how to deal with some problem involving continuations while I watched them play in the tide pools. It felt like I was doing life right. I remember that because I was slightly dismayed at how novel it felt. The good news is that I had more moments like this over the next few years.\n\nIn the summer of 2016 we moved to England.', 'mimetype': 'text/plain', 'start_char_idx': 65997, 'end_char_idx': 66402, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8211530782069452}, {'node': {'id_': 'a98829e7-c59e-4906-9ec8-d1a84ab231e4', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'ac44aacc-cb1f-48db-9581-1ccbc3305a4e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'e99647d69b84b0a13c59268b58406bb00652b41196a27d7b46b56e5d713018ed'}}, 'text': "What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.", 'mimetype': 'text/plain', 'start_char_idx': 2, 'end_char_idx': 373, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8208121222994761}, {'node': {'id_': '3cd01a17-a792-4362-8540-74b68300057f', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': 'c2e92735-3b1c-4d3c-b571-65269e82bbe0', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '0d52c8f8a9f4f6e5d12fdb04b0af0688a87b7cccf5b7337fb595f5e69c4fe2fe'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'f4b10549-f491-4b63-ba16-66a0ae6817b3', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '3779908c352fae99eed78144efa64e8a8a6e91708317acae776ad913bbcc456c'}}, 'text': "Idelle was in New York at least, and there were other people trying to paint there, even though I didn't know any of them.\n\nWhen I got back to New York I resumed my old life, except now I was rich. It was as weird as it sounds. I resumed all my old patterns, except now there were doors where there hadn't been.", 'mimetype': 'text/plain', 'start_char_idx': 38502, 'end_char_idx': 38813, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8174469441429402}], error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 20, 916429), end_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 110019)), pid=54744, tid=6433686), RecordAppCall(call_id='0a2e0a57-48a4-4b6d-ab10-edf0630ac895', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.query_engine, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=14295036320, init_bindings=None), name='retrieve')), RecordAppCallMethod(path=Lens().app.query_engine._retriever, method=Method(obj=Obj(cls=llama_index.core.indices.vector_store.retrievers.retriever.VectorIndexRetriever, id=4348938384, init_bindings=None), name='retrieve'))], args={'str_or_query_bundle': 'What did the author do growing up?'}, rets=[{'node': {'id_': 'ee3bc977-bc7e-4393-9e00-9bd7719f7c1b', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': '538dc6fd-bdfd-42da-b1c2-17ea82554a7d', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '5042ef58fa439b89369e3a10cacb784e1845e881271729a3261403a515d7134b'}, <NodeRelationship.NEXT: '3'>: {'node_id': '62d8a138-8b16-402a-a7cf-2e6dab6633f2', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '0b4f822ad12b07d089691e8fe465cfd1a8eb40b43de3358e4fe2ed3606b289e6'}}, 'text': 'I remember taking the boys to the coast on a sunny day in 2015 and figuring out how to deal with some problem involving continuations while I watched them play in the tide pools. It felt like I was doing life right. I remember that because I was slightly dismayed at how novel it felt. The good news is that I had more moments like this over the next few years.\n\nIn the summer of 2016 we moved to England.', 'mimetype': 'text/plain', 'start_char_idx': 65997, 'end_char_idx': 66402, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8211530782069452}, {'node': {'id_': 'a98829e7-c59e-4906-9ec8-d1a84ab231e4', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'ac44aacc-cb1f-48db-9581-1ccbc3305a4e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'e99647d69b84b0a13c59268b58406bb00652b41196a27d7b46b56e5d713018ed'}}, 'text': "What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.", 'mimetype': 'text/plain', 'start_char_idx': 2, 'end_char_idx': 373, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8208121222994761}, {'node': {'id_': '3cd01a17-a792-4362-8540-74b68300057f', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': 'c2e92735-3b1c-4d3c-b571-65269e82bbe0', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '0d52c8f8a9f4f6e5d12fdb04b0af0688a87b7cccf5b7337fb595f5e69c4fe2fe'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'f4b10549-f491-4b63-ba16-66a0ae6817b3', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '3779908c352fae99eed78144efa64e8a8a6e91708317acae776ad913bbcc456c'}}, 'text': "Idelle was in New York at least, and there were other people trying to paint there, even though I didn't know any of them.\n\nWhen I got back to New York I resumed my old life, except now I was rich. It was as weird as it sounds. I resumed all my old patterns, except now there were doors where there hadn't been.", 'mimetype': 'text/plain', 'start_char_idx': 38502, 'end_char_idx': 38813, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8174469441429402}], error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 20, 869873), end_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 110648)), pid=54744, tid=6433686), RecordAppCall(call_id='7289fbe3-de0c-4585-aef3-33917f8e5e31', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.query_engine, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=14295036320, init_bindings=None), name='retrieve'))], args={'query_bundle': 'What did the author do growing up?'}, rets=[{'node': {'id_': 'ee3bc977-bc7e-4393-9e00-9bd7719f7c1b', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': '538dc6fd-bdfd-42da-b1c2-17ea82554a7d', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '5042ef58fa439b89369e3a10cacb784e1845e881271729a3261403a515d7134b'}, <NodeRelationship.NEXT: '3'>: {'node_id': '62d8a138-8b16-402a-a7cf-2e6dab6633f2', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '0b4f822ad12b07d089691e8fe465cfd1a8eb40b43de3358e4fe2ed3606b289e6'}}, 'text': 'I remember taking the boys to the coast on a sunny day in 2015 and figuring out how to deal with some problem involving continuations while I watched them play in the tide pools. It felt like I was doing life right. I remember that because I was slightly dismayed at how novel it felt. The good news is that I had more moments like this over the next few years.\n\nIn the summer of 2016 we moved to England.', 'mimetype': 'text/plain', 'start_char_idx': 65997, 'end_char_idx': 66402, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8211530782069452}, {'node': {'id_': 'a98829e7-c59e-4906-9ec8-d1a84ab231e4', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'ac44aacc-cb1f-48db-9581-1ccbc3305a4e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'e99647d69b84b0a13c59268b58406bb00652b41196a27d7b46b56e5d713018ed'}}, 'text': "What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.", 'mimetype': 'text/plain', 'start_char_idx': 2, 'end_char_idx': 373, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8208121222994761}, {'node': {'id_': '3cd01a17-a792-4362-8540-74b68300057f', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': 'c2e92735-3b1c-4d3c-b571-65269e82bbe0', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '0d52c8f8a9f4f6e5d12fdb04b0af0688a87b7cccf5b7337fb595f5e69c4fe2fe'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'f4b10549-f491-4b63-ba16-66a0ae6817b3', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '3779908c352fae99eed78144efa64e8a8a6e91708317acae776ad913bbcc456c'}}, 'text': "Idelle was in New York at least, and there were other people trying to paint there, even though I didn't know any of them.\n\nWhen I got back to New York I resumed my old life, except now I was rich. It was as weird as it sounds. I resumed all my old patterns, except now there were doors where there hadn't been.", 'mimetype': 'text/plain', 'start_char_idx': 38502, 'end_char_idx': 38813, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8174469441429402}], error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 20, 824101), end_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 111179)), pid=54744, tid=6433686), RecordAppCall(call_id='92e4a4ba-0436-4746-b45f-941794d40632', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.feedback, method=Method(obj=Obj(cls=trulens_eval.feedback.feedback.Feedback, id=14289375296, init_bindings=None), name='__call__'))], args={'args': ['What did the author do growing up?', 'I remember taking the boys to the coast on a sunny day in 2015 and figuring out how to deal with some problem involving continuations while I watched them play in the tide pools. It felt like I was doing life right. I remember that because I was slightly dismayed at how novel it felt. The good news is that I had more moments like this over the next few years.\n\nIn the summer of 2016 we moved to England.']}, rets=0.2, error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 130574), end_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 573124)), pid=54744, tid=6433903), RecordAppCall(call_id='cd51f080-0c42-47b2-9aec-5c6d0ec2acbc', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.feedback, method=Method(obj=Obj(cls=trulens_eval.feedback.feedback.Feedback, id=14289375296, init_bindings=None), name='__call__'))], args={'args': ['What did the author do growing up?', "What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep."]}, rets=0.7, error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 147557), end_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 574162)), pid=54744, tid=6433904), RecordAppCall(call_id='f35d0781-dec3-4d9a-a3fe-d7e5f499d49d', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.feedback, method=Method(obj=Obj(cls=trulens_eval.feedback.feedback.Feedback, id=14289375296, init_bindings=None), name='__call__'))], args={'args': ['What did the author do growing up?', "Idelle was in New York at least, and there were other people trying to paint there, even though I didn't know any of them.\n\nWhen I got back to New York I resumed my old life, except now I was rich. It was as weird as it sounds. I resumed all my old patterns, except now there were doors where there hadn't been."]}, rets=0.2, error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 161520), end_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 625062)), pid=54744, tid=6433905), RecordAppCall(call_id='257e0aa3-3fc4-4db2-9b05-250fcbb1e719', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.query_engine, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=14295036320, init_bindings=None), name='synthesize')), RecordAppCallMethod(path=Lens().app.query_engine._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.compact_and_refine.CompactAndRefine, id=14250860400, init_bindings=None), name='get_response')), RecordAppCallMethod(path=Lens().app.query_engine._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.refine.Refine, id=14250860400, init_bindings=None), name='get_response')), RecordAppCallMethod(path=Lens().app.query_engine._response_synthesizer._llm, method=Method(obj=Obj(cls=llama_index.llms.openai.base.OpenAI, id=14245284944, init_bindings=None), name='chat'))], args={'_self': {'callback_manager': {'__tru_non_serialized_object': {'cls': {'name': 'CallbackManager', 'module': {'package_name': 'llama_index.core.callbacks', 'module_name': 'llama_index.core.callbacks.base'}, 'bases': None}, 'id': 4348938480, 'init_bindings': None}}, 'system_prompt': None, 'messages_to_prompt': {'__tru_non_serialized_object': {'cls': {'name': 'function', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}, 'id': 14230775392, 'init_bindings': None}}, 'completion_to_prompt': {'__tru_non_serialized_object': {'cls': {'name': 'function', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}, 'id': 14231686816, 'init_bindings': None}}, 'output_parser': None, 'pydantic_program_mode': <PydanticProgramMode.DEFAULT: 'default'>, 'query_wrapper_prompt': None, 'model': 'gpt-3.5-turbo', 'temperature': 0.1, 'max_tokens': None, 'logprobs': None, 'top_logprobs': 0, 'additional_kwargs': {}, 'max_retries': 3, 'timeout': 60.0, 'default_headers': None, 'reuse_client': True, 'api_key': 'sk-...', 'api_base': 'https://api.openai.com/v1', 'api_version': ''}, 'messages': [{'role': <MessageRole.SYSTEM: 'system'>, 'content': "You are an expert Q&A system that is trusted around the world.\nAlways answer the query using the provided context information, and not prior knowledge.\nSome rules to follow:\n1. Never directly reference the given context in your answer.\n2. Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines.", 'additional_kwargs': {}}, {'role': <MessageRole.USER: 'user'>, 'content': "Context information is below.\n---------------------\nfile_path: /Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt\n\nWhat I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: What did the author do growing up?\nAnswer: ", 'additional_kwargs': {}}]}, rets={'message': {'role': <MessageRole.ASSISTANT: 'assistant'>, 'content': 'The author focused on writing and programming outside of school before college. Specifically, the author wrote short stories, which were described as having characters with strong feelings but lacking in plot.', 'additional_kwargs': {}}, 'raw': {'id': 'chatcmpl-9gqs6yDON4VhKRDOBRcUgAJP543EK', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'The author focused on writing and programming outside of school before college. Specifically, the author wrote short stories, which were described as having characters with strong feelings but lacking in plot.', 'role': 'assistant', 'function_call': None, 'tool_calls': None}}], 'created': 1720000042, 'model': 'gpt-3.5-turbo-0125', 'object': 'chat.completion', 'service_tier': None, 'system_fingerprint': None, 'usage': {'completion_tokens': 35, 'prompt_tokens': 229, 'total_tokens': 264}}, 'delta': None, 'logprobs': None, 'additional_kwargs': {}}, error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 821695), end_time=datetime.datetime(2024, 7, 3, 5, 47, 22, 823213)), pid=54744, tid=6433686), RecordAppCall(call_id='d5a88e11-e700-4ddf-88b1-935ec6d0464a', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.query_engine, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=14295036320, init_bindings=None), name='synthesize')), RecordAppCallMethod(path=Lens().app.query_engine._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.compact_and_refine.CompactAndRefine, id=14250860400, init_bindings=None), name='get_response')), RecordAppCallMethod(path=Lens().app.query_engine._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.refine.Refine, id=14250860400, init_bindings=None), name='get_response'))], args={'query_str': 'What did the author do growing up?', 'text_chunks': ["file_path: /Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt\n\nWhat I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep."], 'prev_response': None}, rets='The author focused on writing and programming outside of school before college. Specifically, the author wrote short stories, which were described as having characters with strong feelings but lacking in plot.', error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 770613), end_time=datetime.datetime(2024, 7, 3, 5, 47, 22, 823532)), pid=54744, tid=6433686), RecordAppCall(call_id='5ca98c63-7d95-4b17-9d24-61a557a2428a', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.query_engine, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=14295036320, init_bindings=None), name='synthesize')), RecordAppCallMethod(path=Lens().app.query_engine._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.compact_and_refine.CompactAndRefine, id=14250860400, init_bindings=None), name='get_response'))], args={'query_str': 'What did the author do growing up?', 'text_chunks': ["file_path: /Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt\n\nWhat I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep."]}, rets='The author focused on writing and programming outside of school before college. Specifically, the author wrote short stories, which were described as having characters with strong feelings but lacking in plot.', error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 723704), end_time=datetime.datetime(2024, 7, 3, 5, 47, 22, 823575)), pid=54744, tid=6433686), RecordAppCall(call_id='0c19816c-f709-46ae-8a4a-e097d61d6b87', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app.query_engine, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=14295036320, init_bindings=None), name='synthesize'))], args={'query_bundle': 'What did the author do growing up?', 'nodes': [{'node': {'id_': 'a98829e7-c59e-4906-9ec8-d1a84ab231e4', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'ac44aacc-cb1f-48db-9581-1ccbc3305a4e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'e99647d69b84b0a13c59268b58406bb00652b41196a27d7b46b56e5d713018ed'}}, 'text': "What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.", 'mimetype': 'text/plain', 'start_char_idx': 2, 'end_char_idx': 373, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8208121222994761}]}, rets={'response': 'The author focused on writing and programming outside of school before college. Specifically, the author wrote short stories, which were described as having characters with strong feelings but lacking in plot.', 'source_nodes': [{'node': {'id_': 'a98829e7-c59e-4906-9ec8-d1a84ab231e4', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'ac44aacc-cb1f-48db-9581-1ccbc3305a4e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'e99647d69b84b0a13c59268b58406bb00652b41196a27d7b46b56e5d713018ed'}}, 'text': "What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.", 'mimetype': 'text/plain', 'start_char_idx': 2, 'end_char_idx': 373, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8208121222994761}], 'metadata': {'a98829e7-c59e-4906-9ec8-d1a84ab231e4': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}}}, error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 21, 671166), end_time=datetime.datetime(2024, 7, 3, 5, 47, 22, 823932)), pid=54744, tid=6433686), RecordAppCall(call_id='e8bcc27d-ff8d-433d-96c1-4aec55ab0151', stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=trulens_eval.guardrails.llama.WithFeedbackFilterNodes, id=14295036320, init_bindings=None), name='query'))], args={'query': 'What did the author do growing up?'}, rets={'response': 'The author focused on writing and programming outside of school before college. Specifically, the author wrote short stories, which were described as having characters with strong feelings but lacking in plot.', 'source_nodes': [{'node': {'id_': 'a98829e7-c59e-4906-9ec8-d1a84ab231e4', 'embedding': None, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': '01d1924b-a1ae-4a1b-a728-02c0d2076cdd', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}, 'hash': '1d33ee4442f7d67aae21ae67e94898c45dfefa62763fbe8fa24976ca2c963512'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'ac44aacc-cb1f-48db-9581-1ccbc3305a4e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'e99647d69b84b0a13c59268b58406bb00652b41196a27d7b46b56e5d713018ed'}}, 'text': "What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.", 'mimetype': 'text/plain', 'start_char_idx': 2, 'end_char_idx': 373, 'text_template': '{metadata_str}\n\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\n'}, 'score': 0.8208121222994761}], 'metadata': {'a98829e7-c59e-4906-9ec8-d1a84ab231e4': {'file_path': '/Users/jreini/Desktop/development/trulens/trulens_eval/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-07-03', 'last_modified_date': '2024-07-03'}}}, error=None, perf=Perf(start_time=datetime.datetime(2024, 7, 3, 5, 47, 20, 778856), end_time=datetime.datetime(2024, 7, 3, 5, 47, 22, 824169)), pid=54744, tid=6433686)], feedback_and_future_results=[(FeedbackDefinition(Answer Relevance, selectors={'prompt': Lens().__record__.main_input, 'response': Lens().__record__.main_output}, if_exists=None ), <Future at 0x35116b8f0 state=finished returned FeedbackResult>), (FeedbackDefinition(Context Relevance, selectors={'question': Lens().__record__.main_input, 'context': Lens().__record__.app.query.rets.source_nodes[:].node.text}, if_exists=None ), <Future at 0x350bc6960 state=finished returned FeedbackResult>), (FeedbackDefinition(Groundedness, selectors={'source': Lens().__record__.app.query.rets.source_nodes[:].node.text.collect(), 'statement': Lens().__record__.main_output}, if_exists=None ), <Future at 0x351143f80 state=finished returned FeedbackResult>)], feedback_results=[<Future at 0x35116b8f0 state=finished returned FeedbackResult>, <Future at 0x350bc6960 state=finished returned FeedbackResult>, <Future at 0x351143f80 state=finished returned FeedbackResult>])
tru.run_dashboard()
# The results of the feedback functions can be rertireved from
# `Record.feedback_results` or using the `wait_for_feedback_result` method. The
# results if retrieved directly are `Future` instances (see
# `concurrent.futures`). You can use `as_completed` to wait until they have
# finished evaluating or use the utility method:
for feedback, feedback_result in rec.wait_for_feedback_results().items():
print(feedback.name, feedback_result.result)
# See more about wait_for_feedback_results:
# help(rec.wait_for_feedback_results)
Answer Relevance 0.8 Context Relevance 0.8 Groundedness 1.0
records, feedback = tru.get_records_and_feedback(app_ids=[])
records.head()
app_id | app_json | type | record_id | input | output | tags | record_json | cost_json | perf_json | ts | Groundedness | Answer Relevance | Context Relevance | Groundedness_calls | Answer Relevance_calls | Context Relevance_calls | latency | total_tokens | total_cost | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | LlamaIndex_App1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.core.query_en... | record_hash_d8d7b57e2f4927576f58e09f26469c42 | "What did the author do growing up?" | "The author worked on writing and programming ... | - | {"record_id": "record_hash_d8d7b57e2f4927576f5... | {"n_requests": 2, "n_successful_requests": 3, ... | {"start_time": "2024-07-03T05:47:14.007165", "... | 2024-07-03T05:47:15.029467 | 0.8 | 0.8 | 0.4 | [{'args': {'source': ['I remember taking the b... | [{'args': {'prompt': 'What did the author do g... | [{'args': {'question': 'What did the author do... | 1 | 487 | 0.000713 |
1 | LlamaIndex_App1_Filtered | {"tru_class_info": {"name": "TruLlama", "modul... | WithFeedbackFilterNodes(trulens_eval.guardrail... | record_hash_9f960b879e7fbb4c48a58b1cdfb87b3f | "What did the author do growing up?" | "The author focused on writing and programming... | - | {"record_id": "record_hash_9f960b879e7fbb4c48a... | {"n_requests": 5, "n_successful_requests": 15,... | {"start_time": "2024-07-03T05:47:20.778856", "... | 2024-07-03T05:47:22.824425 | 1.0 | 0.8 | 0.8 | [{'args': {'source': ["What I Worked On\n\nFeb... | [{'args': {'prompt': 'What did the author do g... | [{'args': {'question': 'What did the author do... | 1 | 3537 | 0.005268 |
tru.get_leaderboard(app_ids=[])
Answer Relevance | Context Relevance | Groundedness | latency | total_cost | |
---|---|---|---|---|---|
app_id | |||||
LlamaIndex_App1_Filtered | 0.8 | 0.8 | 1.0 | 2.0 | 0.005268 |
LlamaIndex_App1 | 0.8 | 0.4 | 0.8 | 2.0 | 0.000713 |
Explore in a Dashboard¶
tru.run_dashboard() # open a local streamlit app to explore
# tru.stop_dashboard() # stop if needed
Alternatively, you can run trulens-eval
from a command line in the same folder to start the dashboard.