Truyen Tran - Learning to Remember More with Less Memorization (ICLR 2019 talk)
Authors: Hung T Le, Truyen Tran, Svetha Venkatesh
Speaker: Truyen Tran
Abstract: Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.
Видео Truyen Tran - Learning to Remember More with Less Memorization (ICLR 2019 talk) канала Steven Van Vaerenbergh
Speaker: Truyen Tran
Abstract: Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.
Видео Truyen Tran - Learning to Remember More with Less Memorization (ICLR 2019 talk) канала Steven Van Vaerenbergh
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Suchi Saria: Augmenting Clinical Intelligence with Machine Intelligence (ICLR 2018 invited talk)Evolution of CleverHans (May 15th, 2018)Erik Brynjolfsson: What Can Machine Learning Do? Workforce Implications (ICLR 2018)Machine Learning in Automated Mechanism Design for Pricing and Auctions (ICML 2018 tutorial)Max Welling: Intelligence per Kilowatthour (ICML 2018 invited talk)Using D-ID to create a talking avatar videoPieter Abbeel: Deep Learning for Robotics (NIPS 2017 Keynote)Probabilistic Methods, Applications sessions at NIPS 2017Demo of ChatGPT's visual capabilities (Oct. 2023)Yisong Yue and Hoang M Le: Tutorial on Imitation Learning (ICML 2018 tutorial)Geometric reasoning with ChatGPT and GeoGebra, part 1Deep Learning session at NIPS 2017Michael Unser: Splines and Machine Learning: From classical RKHS methods to DNN (MLSP 2020 keynote)Kristen Grauman: Visual Learning With Unlabeled Video and Look-Around Policies ICLR2018 invited talkYikang Shen: Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (ICLR2019)GitHub Copilot demonstration October 2023 (part 1)Copilot demonstration April 2023Joelle Pineau: Reproducibility, Reusability, and Robustness in Deep Reinforcement Learning ICLR 2018ChatGPT 4 system prompt (December 16, 2023)Eight years of scikit-learn development (Jan. 11th 2018)