Find the BEST RAG Strategy with Domain Specific Evals
Creating custom RAG chunking and embedding strategies with domain specific evaluation experiments
Resources:
Notebook Repo: https://github.com/ALucek/custom-rag-evals/tree/main
Prior Chunking Breakdown: https://youtu.be/Pk2BeaGbcTE?si=pWK6hkXPnJCyb4Ux
ChromaDB Repo: https://github.com/brandonstarxel/chunking_evaluation/tree/main
ChromaDB Colab Notebook: https://colab.research.google.com/drive/1J5ALtDf0_RrswRz2fktjFVeFxe2jbXuJ?usp=sharing#scrollTo=yQiDRQKIf_cl
ChromaDB Research Report: https://research.trychroma.com/evaluating-chunking
Chapters:
00:00 - Why Measure Chunking & Embedding
03:44 - Creating a Custom Chunking Strategy
06:12 - Breaking Down Eval Metrics
08:00 - Metrics: Eval Dataset
09:17 - Metrics: Recall, Precision, IoU
13:27 - General Evals: Describing Test Set
14:32 - General Eval: Process & Running Test
17:58 - General Eval: Embedding Test
19:00 - Running Multiple Evals Across Strategies
20:36 - Multiple Evals: Interpreting Results
23:18 - Domain Specific Dataset Generation & Filtering
27:06 - Running Domain Specific Evals
29:32 - Final Thoughts
#ai #machinelearning #datascience
Видео Find the BEST RAG Strategy with Domain Specific Evals канала Adam Lucek
Resources:
Notebook Repo: https://github.com/ALucek/custom-rag-evals/tree/main
Prior Chunking Breakdown: https://youtu.be/Pk2BeaGbcTE?si=pWK6hkXPnJCyb4Ux
ChromaDB Repo: https://github.com/brandonstarxel/chunking_evaluation/tree/main
ChromaDB Colab Notebook: https://colab.research.google.com/drive/1J5ALtDf0_RrswRz2fktjFVeFxe2jbXuJ?usp=sharing#scrollTo=yQiDRQKIf_cl
ChromaDB Research Report: https://research.trychroma.com/evaluating-chunking
Chapters:
00:00 - Why Measure Chunking & Embedding
03:44 - Creating a Custom Chunking Strategy
06:12 - Breaking Down Eval Metrics
08:00 - Metrics: Eval Dataset
09:17 - Metrics: Recall, Precision, IoU
13:27 - General Evals: Describing Test Set
14:32 - General Eval: Process & Running Test
17:58 - General Eval: Embedding Test
19:00 - Running Multiple Evals Across Strategies
20:36 - Multiple Evals: Interpreting Results
23:18 - Domain Specific Dataset Generation & Filtering
27:06 - Running Domain Specific Evals
29:32 - Final Thoughts
#ai #machinelearning #datascience
Видео Find the BEST RAG Strategy with Domain Specific Evals канала Adam Lucek
artificial intelligence OpenAI AI Gemini Mistral Llama Open Source HuggingFace Machine Learning Deep Learning AI Trends AI Innovations AI Applications AI Tutorial AI Research AI Solutions AI Projects AI Software AI Algorithms Artificial General Intelligence AI Strategy AI Development Multimodal Agent LangChain gpt-4o gpt langsmith fine-tuning RAG data synthetic Learning llm retrieval augmented generation neural evals evaluate domain specific evaluation
Комментарии отсутствуют
Информация о видео
8 января 2025 г. 18:01:12
00:32:30
Другие видео канала
How To Summarize & Chunk Technical Documents with AI Language Models
How AI Creates Images/Videos/Audio - Diffusion Models Explained
Building a Thinking Machine: My Gemma 2 9B Reflection Agent
Why Every AI Developer Should Learn Model Context Protocol (MCP)
Fine Tuning OpenAI Models Walkthrough - How and Why
Model Output Comparison with GPT/Gemini/Mistral/Llama2 - PulseRead Followup
Making YouTube Videos With AI! - How to Use LLM & ATT Models to Make Engaging Content
LLM Model Output Comparison - Mistral Large/Gemma 7b/Llama 70b/Claude 3 Opus & More Pt.2
Master Reinforcement Learning With These 3 Projects
Large Multimodal Models Are The Future - Text/Vision/Audio in LLMs
How To Easily Run & Use LLMs Locally - Ollama & LangChain Integration
Improving RAG Retrieval by 60% with Fine-Tuned Embeddings
Build Your Own Finance LLM for FREE with SEC Data
Does Fine Tuning Embedding Models Improve RAG?
Quantizing LLMs - How & Why (8-Bit, 4-Bit, GGUF & More)
BERT: The Most Used AI Model You Haven't Heard Of
Stop Prompt Engineering! Program Your LLMs with DSPy
Breaking Down & Testing FIVE LLM Agent Architectures - (Reflexion, LATs, P&E, ReWOO, LLMCompiler)
The BEST Way to Chunk Text for RAG
Multimodal RAG!? - Pushing the Boundaries of AI
I Trained an LLM to Think Deeper (Here's How)