#035 Christmas Community Edition!

Welcome to the Christmas special community edition of MLST! We discuss some recent and interesting papers from Pedro Domingos (are NNs kernel machines?), Deepmind (can NNs out-reason symbolic machines?), Anna Rodgers - When BERT Plays The Lottery, All Tickets Are Winning, Prof. Mark Bishop (even causal methods won't deliver understanding), We also cover our favourite bits from the recent Montreal AI event run by Prof. Gary Marcus (including Rich Sutton, Danny Kahneman and Christof Koch). We respond to a reader mail on Capsule networks. Then we do a deep dive into Type Theory and Lambda Calculus with community member Alex Mattick. In the final hour we discuss inductive priors and label information density with another one of our discord community members.

Panel: Dr. Tim Scarfe, Yannic Kilcher, Alex Stenlake, Dr. Keith Duggar

Enjoy the show and don't forget to subscribe!

00:00:00 Welcome to Christmas Special!
00:00:44 SoTa meme
00:01:30 Happy Christmas!
00:03:11 Paper -- DeepMind - Outperforming neuro-symbolic models with NNs (Ding et al)
00:08:57 What does it mean to understand?
00:17:37 Paper - Prof. Mark Bishop Artificial Intelligence is stupid and causal reasoning
wont fix it
00:25:39 Paper -- Pedro Domingos - Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
00:31:07 Paper - Bengio - Inductive Biases for Deep Learning of Higher-Level Cognition
00:32:54 Anna Rodgers - When BERT Plays The Lottery, All Tickets Are Winning
00:37:16 Montreal AI event - Gary Marcus on reasoning
00:40:37 Montreal AI event -- Rich Sutton on universal theory of AI
00:49:45 Montreal AI event -- Danny Kahneman, System 1 vs 2 and Generative Models ala free energy principle
01:02:57 Montreal AI event -- Christof Koch - Neuroscience is hard
01:10:55 Markus Carr -- reader letter on capsule networks
01:13:21 Alex response to Marcus Carr
01:22:06 Type theory segment -- with Alex Mattick from Discord
01:24:45 Type theory segment -- What is Type Theory
01:28:12 Type theory segment -- Difference between functional and OOP languages
01:29:03 Type theory segment -- Lambda calculus
01:30:46 Type theory segment -- Closures
01:35:05 Type theory segment -- Term rewriting (confluency and termination)
01:42:02 MType theory segment -- eta term rewritig system - Lambda Calculus
01:54:44 Type theory segment -- Types / semantics
02:06:26 Type theory segment -- Calculus of constructions
02:09:27 Type theory segment -- Homotopy type theory
02:11:02 Type theory segment -- Deep learning link
02:17:27 Jan from Discord segment -- Chrome MRU skit
02:18:56 Jan from Discord segment -- Inductive priors (with XMaster96/Jan from Discord)
02:37:59 Jan from Discord segment -- Label information density (with XMaster96/Jan from Discord)
02:55:13 Outro

Pod version: https://anchor.fm/machinelearningstreettalk/episodes/035-Christmas-Community-Edition-eo9b2s

Marcus Karr email links (capsule networks);

https://ai.facebook.com/blog/end-to-end-object-detection-with-transformers/
https://science.sciencemag.org/content/sci/358/6368/eaag2612/F3.large.jpg?width=800&height=600&carousel=1
https://science.sciencemag.org/content/358/6368/eaag2612/tab-figures-data
http://akosiorek.github.io/ml/2019/06/23/stacked_capsule_autoencoders.html
https://www.youtube.com/watch?v=UX8OubxsY8w&feature=youtu.be

Alex Mattick supporting PDF file on Type Theory discussion: https://drive.google.com/file/d/18LC9rSKU7RUougGn8EpuMDqlipO00AB0/view?usp=sharing

Papers discussed:

Every Model Learned by Gradient Descent Is Approximately a Kernel Machine (Pedro Domingos)
https://arxiv.org/pdf/2012.00152.pdf

Inductive Biases for Deep Learning of Higher-Level Cognition (Goyal and Bengio)
https://arxiv.org/pdf/2011.15091.pdf

Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures (Ding et al @ Deepmind)
https://arxiv.org/pdf/2012.08508.pdf

Deep learning for symbollic mathematics (Lample et al @ FAIR)
https://arxiv.org/pdf/1912.01412.pdf

Artificial Intelligence is stupid and causal reasoning wont fix it (J. Mark Bishop)
https://arxiv.org/pdf/2008.07371.pdf

Видео #035 Christmas Community Edition! канала Machine Learning Street Talk