Back

Speaker "Joshua Reini" Details Back

 

Topic

Evaluating and Tracking LLM Experiments: Building Better LLM Apps with TruLens

Abstract

Building LLM apps that combine powerful LLMs with vector databases, agents and more? If you’re developing with a framework like LlamaIndex or LangChain; an LLM like one from OpenAI or Hugging Face; or using a vector database from Pinecone or Chroma, you will want to learn how to measure the performance and quality of your LLM-based applications using feedback functions. This session explores what feedback functions are, how to use them, and why they can make all the difference.
Who is this presentation for?
LLM application developers.
Prerequisite knowledge:
LLM app development basics.
What you'll learn?
Building LLM apps that combine powerful LLMs with vector databases, agents and more? If you’re developing with a framework like LlamaIndex or LangChain; an LLM like one from OpenAI or Hugging Face; or using a vector database from Pinecone or Chroma, you will want to learn how to measure the performance and quality of your LLM-based applications using feedback functions. This session explores what feedback functions are, how to use them, and why they can make all the difference. This session covers: The challenges with LLM app development today What’s a feedback function and how does it work? How to put feedback functions to good use as you are developing LLM apps Tracking performance, quality, and cost across LLM app versions Demo of TruLens for LLM Apps, an open source software toolkit that uses feedback functions for evaluating LLM apps

Profile

Josh is a core contributor to open-source TruLens and the founding Developer Relations Data Scientist at TruEra where he is responsible for education initiatives and nurturing a thriving community of AI Quality practitioners. Prior to TruEra, Josh delivered end-to-end data and machine learning and solutions to clients including the Department of State and the Walter Reed National Military Medical Center. During his time at Walter Reed, he was published in the Journal of Telemedicine and e-Health as the lead statistician for a clinical trial involving a novel heart rate device. Josh also worked in product management at Geico and has a Master’s degree in Economics from the University of Georgia.