Yikang Shen: Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (ICLR2019)
Speaker: Yikang Shen
Paper: Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
Authors: Yikang Shen, Shawn Tan, Alessandro Sordoni, Aaron Courville
In general, natural language is governed by a tree structure: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). This is a strict hierarchy: when a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM allows different neurons to track information at different time scales, the architecture does not impose a strict hierarchy. This paper proposes to add such a constraint to the system by ordering the neurons; a vector of "master" input and forget gates ensure that when a given unit is updated, all of the units that follow it in the ordering are also updated. To this end, we propose a new RNN unit: ON-LSTM, which achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.
Presented at ICLR 2019
Видео Yikang Shen: Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (ICLR2019) канала Steven Van Vaerenbergh
Paper: Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
Authors: Yikang Shen, Shawn Tan, Alessandro Sordoni, Aaron Courville
In general, natural language is governed by a tree structure: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). This is a strict hierarchy: when a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM allows different neurons to track information at different time scales, the architecture does not impose a strict hierarchy. This paper proposes to add such a constraint to the system by ordering the neurons; a vector of "master" input and forget gates ensure that when a given unit is updated, all of the units that follow it in the ordering are also updated. To this end, we propose a new RNN unit: ON-LSTM, which achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.
Presented at ICLR 2019
Видео Yikang Shen: Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (ICLR2019) канала Steven Van Vaerenbergh
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Suchi Saria: Augmenting Clinical Intelligence with Machine Intelligence (ICLR 2018 invited talk)Evolution of CleverHans (May 15th, 2018)Erik Brynjolfsson: What Can Machine Learning Do? Workforce Implications (ICLR 2018)Machine Learning in Automated Mechanism Design for Pricing and Auctions (ICML 2018 tutorial)Max Welling: Intelligence per Kilowatthour (ICML 2018 invited talk)Using D-ID to create a talking avatar videoPieter Abbeel: Deep Learning for Robotics (NIPS 2017 Keynote)Probabilistic Methods, Applications sessions at NIPS 2017Demo of ChatGPT's visual capabilities (Oct. 2023)Yisong Yue and Hoang M Le: Tutorial on Imitation Learning (ICML 2018 tutorial)Truyen Tran - Learning to Remember More with Less Memorization (ICLR 2019 talk)Geometric reasoning with ChatGPT and GeoGebra, part 1Deep Learning session at NIPS 2017Michael Unser: Splines and Machine Learning: From classical RKHS methods to DNN (MLSP 2020 keynote)Kristen Grauman: Visual Learning With Unlabeled Video and Look-Around Policies ICLR2018 invited talkGitHub Copilot demonstration October 2023 (part 1)Copilot demonstration April 2023Joelle Pineau: Reproducibility, Reusability, and Robustness in Deep Reinforcement Learning ICLR 2018ChatGPT 4 system prompt (December 16, 2023)Eight years of scikit-learn development (Jan. 11th 2018)