GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
Link: https://arxiv.org/abs/2201.08860
Abstract: Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains an open question how to effectively fuse and reason over the KG representations and the language context, which provides situational constraints and nuances. In this work, we propose GreaseLM, a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains demonstrate that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.
Видео GreaseLM: Graph REASoning Enhanced Language Models for Question Answering канала The NLP Lab
Abstract: Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains an open question how to effectively fuse and reason over the KG representations and the language context, which provides situational constraints and nuances. In this work, we propose GreaseLM, a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains demonstrate that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.
Видео GreaseLM: Graph REASoning Enhanced Language Models for Question Answering канала The NLP Lab
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
CTRLsum: Towards Generic Controllable Text SummarizationJukebox: A Generative Model for Music | NLP Journal ClubRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks | NLP journal clubMemorizing TransformersAnswering Complex Open-Domain Questions with Multi-Hop Dense Retrieval | NLP Journal ClubPitfalls of Static Language Modelling | NLP Journal ClubHow Close is ChatGPT to Human Experts?Best General-purpose NLP Libraries to Use in 2021Who will Win the Large Language Model App Race? with @SlatorLearning to Reason and Memorize with Self-Notes | Paper summaryFalcon LLM: the Best Open-source LLM Available at the MomentTree of Thoughts: Deliberate Problem Solving with Large Language Models | Paper summaryWhen to use a large language model? 4 points to consider in 2023.QURIOUS: Question Generation Pretraining for Text Generation | NLP Journal ClubA Distributional Approach to Controlled Text Generation | NLP Journal ClubREALM: Retrieval-Augmented Language Model Pre-Training | NLP Journal ClubMulti-scale Transformer Language Models*Paper summary* ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte ModelsMirror-Generative Neural Machine Translation | NLP Journal ClubDeep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT | NLP Journal Club