Production-Ready AI Systems: Security, Evaluation & Data Platforms

Modern AI systems require more than powerful models—they require security, evaluation, governance, and continuous improvement. This session combines lessons from production AI agent security with real-world LLM evaluation and fine-tuning workflows.

Topics may include prompt injection, tool abuse, memory poisoning, defense-in-depth architectures, custom evaluation frameworks, Azure OpenAI fine- tuning, and practical engineering lessons learned from deploying AI-powered systems.

Key Takeaways:
- Understand security challenges in AI agents
- Learn practical defense patterns for production AI
- Explore LLM evaluation methodologies
- Understand fine-tuning workflows using Azure OpenAI
- Apply production engineering best practices to AI systems

📌 This is part of a series, learn more here: https://aka.ms/ProdReadySystems/series

00:00 Intro & Housekeeping
01:50 Securing the AI Stack: Why AI Security Matters
05:00 The Four-Layer AI Security Framework
06:48 Layer 1: Model Security (Prompt Injection, Jailbreaks & Output Hijacking)
10:55 Layer 2: Agent & Application Security
14:09 Layer 3: MCP & Tool Security Risks
16:06 Layer 4: Infrastructure Security Essentials
17:57 Defense in Depth & Security Checklist
19:42 Resources & Final Takeaways on AI Security
20:43 LLM-Driven Merge Conflict Resolution Introduction
22:21 Why Merge Conflicts Are Still a Major Developer Challenge
28:23 Fine-Tuning LLMs for Merge Conflict Resolution
29:40 Key Fine-Tuning Insights & Lessons Learned
33:31 Evaluating Merge Resolution Models
34:42 Python Tips, Structured Outputs & Development Best Practices
39:00 Handling Large Files & Token Limits
40:46 Q&A Transition
42:07 Agent Security in Practice: Real-World Risks & Attacks
43:26 Hugging Face Security Incident Case Study
47:44 OWASP Top Risks for LLM Applications
54:05 Live Demo: Testing & Defending Against Prompt Injection
56:17 Excessive Permissions, Data Exposure & Supply Chain Risks
59:30 Token Abuse, Rate Limiting & Secure Agent Design
01:01:09 Evaluations, Red Teaming & Reliability Testing
01:03:09 Slides, Q&A & Resources
01:03:46 Closing Remarks & Event Survey

[eventID:27335]

Видео Production-Ready AI Systems: Security, Evaluation & Data Platforms канала Microsoft Reactor