OpenAI GPT-3: Language Models are Few-Shot Learners
**ERRATA**: Open AI/GPT-3 DOES NOT USE Microsoft's ZeRO/DeepSpeed for training
Discord: https://discord.gg/4H8xxDF
In this episode of Machine Learning Street Talk, Tim Scarfe, Yannic Kilcher and Connor Shorten discuss their takeaways from OpenAI’s GPT-3 language model. OpenAI trained a 175 BILLION parameter autoregressive language model. The paper demonstrates how self-supervised language modelling at this scale can perform many downstream tasks without fine-tuning.
00:00:00 Intro
00:00:54 ZeRO1+2 (model + Data parallelism) [GPT-3 DOES *NOT* USE THIS] (Connor)
00:03:17 Recent history of NLP (Tim)
00:06:04 Yannic "Light-speed" Kilcher's brief overview of GPT-3
00:14:25 Reviewing Yannic's YT comments on his GPT-3 video (Tim)
00:20:26 Main show intro
00:23:03 Is GPT-3 reasoning?
00:28:15 Architecture discussion and autoregressive (GPT*) vs denoising autoencoder (BERT)
00:36:18 Utility of GPT-3 in industry
00:43:03 Can GPT-3 do math? (reasoning/system 1/system 2)
00:51:03 Generalisation
00:56:48 Esoterics of language models
00:58:46 Architectural trade-offs
01:07:37 Memorization machines and intepretability
01:17:16 Nearest neighbour probes / watermarks
01:20:03 YouTube comments on GPT-3 video
01:21:50 GPT-3 news article generation issue
01:27:36 Sampling data for language models / bias / fairness / politics
01:51:12 Outro
These paradigms of task adaptation are divided into zero, one, and few shot learning. Zero-shot learning is a very extreme case where we expect a language model to perform a task such as sentiment classification or extractive question answering, without any additional supervision. One and Few-shot learning provide some examples to the model. However, GPT-3s definition of this diverges a bit from the conventional literature. GPT-3 provides one and few-shot examples in the form of “In-Context Learning”. Instead of fine-tuning the model on a few examples, the model has to use the input to infer the downstream task. For example, the GPT-3 transformer has an input sequence of 2048 tokens, so demonstrations of a task such as yelp sentiment reviews, would have to fit in this input sequence as well as the new review.
**ERRATA-continued** It has come to our attention that there was a serious factual error in our video -- GPT-3 DOES NOT USE Microsoft's ZeRO/ZeRO2 or DeepSpeed for training and there is no reference to this in either their blog post or paper. We are really sorry about this mistake and will be more careful to fact-check in future.
Thanks for watching! Please Subscribe!
Paper Links:
GPT-3: https://arxiv.org/abs/2005.14165
#machinelearning #naturallanguageprocessing #deeplearning #gpt3
Видео OpenAI GPT-3: Language Models are Few-Shot Learners канала Machine Learning Street Talk
Discord: https://discord.gg/4H8xxDF
In this episode of Machine Learning Street Talk, Tim Scarfe, Yannic Kilcher and Connor Shorten discuss their takeaways from OpenAI’s GPT-3 language model. OpenAI trained a 175 BILLION parameter autoregressive language model. The paper demonstrates how self-supervised language modelling at this scale can perform many downstream tasks without fine-tuning.
00:00:00 Intro
00:00:54 ZeRO1+2 (model + Data parallelism) [GPT-3 DOES *NOT* USE THIS] (Connor)
00:03:17 Recent history of NLP (Tim)
00:06:04 Yannic "Light-speed" Kilcher's brief overview of GPT-3
00:14:25 Reviewing Yannic's YT comments on his GPT-3 video (Tim)
00:20:26 Main show intro
00:23:03 Is GPT-3 reasoning?
00:28:15 Architecture discussion and autoregressive (GPT*) vs denoising autoencoder (BERT)
00:36:18 Utility of GPT-3 in industry
00:43:03 Can GPT-3 do math? (reasoning/system 1/system 2)
00:51:03 Generalisation
00:56:48 Esoterics of language models
00:58:46 Architectural trade-offs
01:07:37 Memorization machines and intepretability
01:17:16 Nearest neighbour probes / watermarks
01:20:03 YouTube comments on GPT-3 video
01:21:50 GPT-3 news article generation issue
01:27:36 Sampling data for language models / bias / fairness / politics
01:51:12 Outro
These paradigms of task adaptation are divided into zero, one, and few shot learning. Zero-shot learning is a very extreme case where we expect a language model to perform a task such as sentiment classification or extractive question answering, without any additional supervision. One and Few-shot learning provide some examples to the model. However, GPT-3s definition of this diverges a bit from the conventional literature. GPT-3 provides one and few-shot examples in the form of “In-Context Learning”. Instead of fine-tuning the model on a few examples, the model has to use the input to infer the downstream task. For example, the GPT-3 transformer has an input sequence of 2048 tokens, so demonstrations of a task such as yelp sentiment reviews, would have to fit in this input sequence as well as the new review.
**ERRATA-continued** It has come to our attention that there was a serious factual error in our video -- GPT-3 DOES NOT USE Microsoft's ZeRO/ZeRO2 or DeepSpeed for training and there is no reference to this in either their blog post or paper. We are really sorry about this mistake and will be more careful to fact-check in future.
Thanks for watching! Please Subscribe!
Paper Links:
GPT-3: https://arxiv.org/abs/2005.14165
#machinelearning #naturallanguageprocessing #deeplearning #gpt3
Видео OpenAI GPT-3: Language Models are Few-Shot Learners канала Machine Learning Street Talk
Показать
Комментарии отсутствуют
Информация о видео
7 июня 2020 г. 4:41:12
01:51:48
Другие видео канала
![GPT-3: Language Models are Few-Shot Learners (Paper Explained)](https://i.ytimg.com/vi/SY5PvZrJhLE/default.jpg)
![Sam Altman (CEO, OpenAI) Talks About Artificial Intelligence at Big Compute 20 Tech Conference](https://i.ytimg.com/vi/0TRtSk-ufu0/default.jpg)
![#51 Francois Chollet - Intelligence and Generalisation](https://i.ytimg.com/vi/J0p_thJJnoo/default.jpg)
![#046 The Great ML Stagnation (Mark Saroufim and Dr. Mathew Salvaris)](https://i.ytimg.com/vi/BwhBtvCNwxo/default.jpg)
![Elon Musk on Artificial Intelligence](https://i.ytimg.com/vi/H15uuDMqDK0/default.jpg)
![](https://i.ytimg.com/vi/10NOIZGgQUY/default.jpg)
![Transformer Neural Networks - EXPLAINED! (Attention is all you need)](https://i.ytimg.com/vi/TQQlZhbC5ps/default.jpg)
![14 Cool Apps Built on OpenAI's GPT-3 API](https://i.ytimg.com/vi/G6Z_S6hs29s/default.jpg)
![How to learn any language easily | Matthew Youlden | TEDxClapham](https://i.ytimg.com/vi/Yr_poW-KK1Q/default.jpg)
![#036 - Max Welling: Quantum, Manifolds & Symmetries in ML](https://i.ytimg.com/vi/mmDw5glry9w/default.jpg)
![#031 WE GOT ACCESS TO GPT-3! (With Gary Marcus, Walid Saba and Connor Leahy)](https://i.ytimg.com/vi/iccd86vOz3w/default.jpg)
![#48 Machine Learning Security - Andy Smith](https://i.ytimg.com/vi/7Tz-3S4lypI/default.jpg)
![Learning from dirty jobs | Mike Rowe](https://i.ytimg.com/vi/IRVdiHu1VCc/default.jpg)
![#52 - Adversarial Examples Beyond Security (Hadi Salman, MIT)](https://i.ytimg.com/vi/_eHRICHlg1k/default.jpg)
![GPT-3 - Language Models are Few-Shot Learners | Paper Explained](https://i.ytimg.com/vi/fVt387VZJe8/default.jpg)
![039 - Lena Voita - NLP](https://i.ytimg.com/vi/Q0kN_ZHHDQY/default.jpg)
![Attention Is All You Need](https://i.ytimg.com/vi/iDulhoQ2pro/default.jpg)
![Sayak Paul](https://i.ytimg.com/vi/LUx8KyzEzSE/default.jpg)
![I used the GPT-2 AI to respond to my YouTube comments](https://i.ytimg.com/vi/3mUnEywtdZY/default.jpg)
![CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://i.ytimg.com/vi/1MprzvYNpY8/default.jpg)