RoBERTa: A Robustly Optimized BERT Pretraining Approach
This paper shows that the original BERT model, if trained correctly, can outperform all of the improvements that have been proposed lately, raising questions about the necessity and reasoning behind these.
Abstract:
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.
Authors: Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov
https://arxiv.org/abs/1907.11692
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Minds: https://www.minds.com/ykilcher
BitChute: https://www.bitchute.com/channel/10a5ui845DOJ/
Видео RoBERTa: A Robustly Optimized BERT Pretraining Approach канала Yannic Kilcher
Abstract:
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.
Authors: Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov
https://arxiv.org/abs/1907.11692
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Minds: https://www.minds.com/ykilcher
BitChute: https://www.bitchute.com/channel/10a5ui845DOJ/
Видео RoBERTa: A Robustly Optimized BERT Pretraining Approach канала Yannic Kilcher
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingXLNet: Generalized Autoregressive Pretraining for Language UnderstandingGPT-3: Language Models are Few-Shot Learners (Paper Explained)Should you switch from BERT to ALBERT?Multilingual BERT - Part 1 - Intro and ConceptsFixMatch: Simplifying Semi-Supervised Learning with Consistency and ConfidenceManifold Mixup: Better Representations by Interpolating Hidden StatesNLP | Fine Tuning BERT to perform Spam ClassificationSentence Transformers: Sentence-BERT - Sentence Embeddings using Siamese BERT-Networks | arXiv demoBERT Model Architectures For Semantic SimilarityFNet: Mixing Tokens with Fourier Transforms (Machine Learning Research Paper Explained)Transformer Neural Networks - EXPLAINED! (Attention is all you need)[Rant] Can AI read your emotions? (No, but ...)[Classic] Deep Residual Learning for Image Recognition (Paper Explained)Expire-Span: Not All Memories are Created Equal: Learning to Forget by Expiring (Paper Explained)Topic Modeling with BERT Transformers - EXPLAINED!Decision Transformer: Reinforcement Learning via Sequence Modeling (Research Paper Explained)DDPM - Diffusion Models Beat GANs on Image Synthesis (Machine Learning Research Paper Explained)Group Normalization (Paper Explained)