How Top AI Product Managers Evaluate Products | Ep36

Subscribe to our newsletter: https://dataneighbor.substack.com/
DNP Swag Store: https://shop.dataneighbor.com/

AI is changing product management, from how teams prototype to how they measure success. In this episode of the Data Neighbor Podcast, we’re joined by Aman Khan, Head of Product at Arize AI (LLM evaluation & observability). Aman breaks down the three emerging AI PM archetypes (AI-native PM, AI platform PM, and AI-powered PM), how to move from “vibe coding” to eval-driven development (EDD), and why aligning evals to business outcomes matters more than any single accuracy score. He also shares hard-won tactics for handling subjectivity in LLM outputs, setting user expectations in UX, and deciding when rigor can (and can’t) slow down speed.

In this episode, you’ll learn:

-The three ways AI shows up in PM work—and how those roles are converging.
-A practical ladder from “vibe checks” to EDD (evals in dev & production), including LLM-as-a-judge and when to trust it.
-How to tie evals to business metrics (trust, value, speed) and resolve “good eval, bad outcome” conflicts.
-UX patterns for long-running agent tasks (progress, ETAs, checkpoints) that preserve trust.
-Where AI coding tools help most (and least) across engineers, PMs, and data teams.

Connect with Aman Khan:
LinkedIn: https://www.linkedin.com/in/amanberkeley/
🌐 Website: https://amank.ai
🏢 Arize AI: https://arize.com/
Arize AI

Connect with Shane, Sravya, and Hai (let us know YouTube sent you!):
👉 Shane Butler: https://linkedin.openinapp.co/b02fe
👉 Sravya Madipalli: https://linkedin.openinapp.co/9be8c
👉 Hai Guan: https://linkedin.openinapp.co/4qi1r

Chapters:
00:00 Introduction to AI Product Management
05:03 The Role of AI in Product Development
08:30 Understanding Evals in AI Applications
11:53 Workflows in AI Product Management
15:23 Evaluating AI Performance and Business Outcomes
18:57 The Future of AI Tools and Productivity
22:16 Navigating Subjectivity in AI Responses
32:50 Trust and AI: Navigating Client Relationships
33:39 Evaluating AI: Best Practices and Principles
34:49 Understanding User Needs: Trust, Value, and Speed
36:05 Balancing Speed and Rigor in User Experience
37:07 The Importance of User Expectations in AI Responses
38:46 Anticipating User Needs: Proactive AI Interactions
40:32 Innovative Task Management with AI
41:46 The Evolution of AI in Team Dynamics
44:12 The Impact of AI on Human Patience and Productivity
47:03 Challenges in Data Teams: Trust and Governance
50:24 Insights from Data: The Role of Collaboration
51:56 The Future of Evaluations in AI Systems
55:24 Closing Thoughts

#aiproductmanagement #aievals #llmobservability #productmanagement #datascience #mlops #aiagents #evaluation #productstrategy #dataneighbor #arizeai #llms

Видео How Top AI Product Managers Evaluate Products | Ep36 канала Data Neighbor Podcast