[Own work] MM-SHAP to measure modality contributions

Today we present our own work on MM-SHAP which measures how much a multimodal encoder uses each modality. Ah, what is multimodality again? 👉 https://youtu.be/jReaoJWdO78

📜 Parcalabescu, Letitia, and Anette Frank. "MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks." arXiv preprint arXiv:2212.08158 (2022). https://arxiv.org/abs/2212.08158

📺 VeLO trained optimizers: https://youtu.be/9a6PQJxzUpM
📺 Watermarking Large Language models: https://youtu.be/-vToUx5SDW4
📺 Paella text-to-image diffusion model: https://youtu.be/6zeLSANd41k

❓Check out our #MachineLearning Quiz Questions: https://www.youtube.com/c/AICoffeeBreak/community
➡️ AI Coffee Break Merch! 🛍️ https://aicoffeebreak.creator-spring.com/

Outline:
00:00 Paper for ACL 2023 Toronto
00:24 Vision and Language Transformers
01:05 Unimodal collapse
02:46 MM-SHAP
04:21 Not all models use modalities to the same extent
06:02 Outro and Final words

Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏
Dres. Trost GbR, Siltax, Edvard Grødem, Vignesh Valliappan, Mutual Information, Mike Ton

▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕
Patreon: https://www.patreon.com/AICoffeeBreak
Ko-fi: https://ko-fi.com/aicoffeebreak
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

🔗 Links:
AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community
Twitter: https://twitter.com/AICoffeeBreak
Reddit: https://www.reddit.com/r/AICoffeeBreak/
YouTube: https://www.youtube.com/AICoffeeBreak

#AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research​

Video editing: Nils Trost

Видео [Own work] MM-SHAP to measure modality contributions канала AI Coffee Break with Letitia