Training learned optimizers: VeLO paper EXPLAINED

Why tune optimizers hyperparameters (Adam) by hand, when one can train a neural network to behave like an optimizer and dynamically find the best update for your neural network’s weights?
In this video, we explain the work on VeLO to train an optimizer from data from previous training runs.
► Sponsor: Cohere 👉 https://t1p.de/22srn

Check out our daily #MachineLearning Quiz Questions: https://www.youtube.com/c/AICoffeeBreak/community

📜 VeELO paper: Metz, Luke, James Harrison, C. Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal et al. "VeLO: Training Versatile Learned Optimizers by Scaling Up." https://arxiv.org/abs/2211.09760

Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏
Dres. Trost GbR, Siltax, Edvard Grødem, Vignesh Valliappan, Mutual Information, Mike Ton

Outline:
00:00 VeLO optimizer without any hyperparameters
01:13 Cohere [Sponsor]
02:27 What are optimizers?
04:37 VeLO idea and training data
06:43 VeLO model and training
10:15 What can VeLO do?
11:52 Limitations of VeLO
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕
Patreon: https://www.patreon.com/AICoffeeBreak
Ko-fi: https://ko-fi.com/aicoffeebreak
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

🔗 Links:
AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community
Twitter: https://twitter.com/AICoffeeBreak
Reddit: https://www.reddit.com/r/AICoffeeBreak/
YouTube: https://www.youtube.com/AICoffeeBreak

#AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research​

Video editing: Nils Trost
Music 🎵 : Hey There - half.cool

Видео Training learned optimizers: VeLO paper EXPLAINED канала AI Coffee Break with Letitia