Load Testing LLMs on Databricks with AIPerf

In this video we utilize AIPerf to load test Databricks hosted LLMs. With Databricks managed clusters and serverless offerings along with the LLM specific metrics AIPerf provides, you can test your LLMs at scale.

Video Resources
- Notebook Code: https://github.com/RamVegiraju/llm-eval-testing-harness
- AIPerf: https://github.com/ai-dynamo/aiperf
- DBX FM API: https://docs.databricks.com/aws/en/machine-learning/foundation-model-apis/

Timestamps
0:00 Introduction
1:30 LLM Specific Metrics
8:50 AIPerf Framework
11:15 Hands-On

#databricks #modelserving #llms #agents

Видео Load Testing LLMs on Databricks with AIPerf канала Ram Vegiraju