Cross-Task Generalization via Natural Language Crowdsourcing Instructions
This video explains the paper "https://arxiv.org/abs/2104.08773".
Abstract: Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.
Видео Cross-Task Generalization via Natural Language Crowdsourcing Instructions канала Allen Institute for AI
Abstract: Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.
Видео Cross-Task Generalization via Natural Language Crowdsourcing Instructions канала Allen Institute for AI
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
![Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models](https://i.ytimg.com/vi/_Q59o0f_HC8/default.jpg)
![Visual Reaction](https://i.ytimg.com/vi/iyAoPuHxvYs/default.jpg)
![Horacio Saggion: Mining and Enriching Scientific Text Collections](https://i.ytimg.com/vi/DB6DcKYlC4w/default.jpg)
![Ajay Nagesh: Exploring Relational Features and Learning](https://i.ytimg.com/vi/LzcIUIFlvSA/default.jpg)
![Learning for Never-before-seen Biomedicine](https://i.ytimg.com/vi/GjL3jjQFVh0/default.jpg)
![Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE](https://i.ytimg.com/vi/gapJa67kaKc/default.jpg)
![Rishabh Iyer: Submodular Optimization and Data Summarization with Applications to Computer Vision](https://i.ytimg.com/vi/LEpS0iVdD4o/default.jpg)
![Kevin Gimpel: From Paraphrase Modeling to Controlled Generation](https://i.ytimg.com/vi/W-q6iTfWxM4/default.jpg)
![Applied AI in High-Expertise Settings, or Curation as Programming](https://i.ytimg.com/vi/zmeLQiO_P1M/default.jpg)
![From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project | AI2](https://i.ytimg.com/vi/CR3aICkhCJM/default.jpg)
![When Not to Trust Language Models: Investigating Effectiveness of Parametric&Non-Parametric Memories](https://i.ytimg.com/vi/hJbxW0xct2E/default.jpg)
![Visual Room Rearrangement (CVPR 2021)](https://i.ytimg.com/vi/1APxaOC9U-A/default.jpg)
![Adapting to Long Tail Domains: A Case Study in Clinical Information | AI2](https://i.ytimg.com/vi/cc1SIr1Heaw/default.jpg)
![Kenneth D. Forbus: Multimodal Science Learning](https://i.ytimg.com/vi/rzS-1fZ26G8/default.jpg)
![Daniel Khashabi - Natural Language Understanding with Indirect Supervision](https://i.ytimg.com/vi/__h_iApLbys/default.jpg)
![Jesse Dodge: Open Loop Hyperparameter Optimization and Determinantal Point Processes](https://i.ytimg.com/vi/el_DbbqXuQY/default.jpg)
![Dr. Asma Ben Abacha: Medical Question Answering](https://i.ytimg.com/vi/Fjsz5Giw9rs/default.jpg)
![Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for System Improvement](https://i.ytimg.com/vi/c5j_tWsENFg/default.jpg)
![Robot Learning by Understanding Egocentric Videos](https://i.ytimg.com/vi/4WznUQvDQEw/default.jpg)
![Explaining Answers with Entailment Trees](https://i.ytimg.com/vi/QPSZQYA1RmA/default.jpg)