A brief history of the Transformer architecture in NLP

🏛️ The Transformer architecture has revolutionized Natural Language Processing, being capable to beat the state-of-the-art on overwhelmingly numerous tasks! Check out this video for a brief history of the Transformer development.

Related video: How do we check if a neural network has learned a specific phenomenon? https://youtu.be/fL22NAtMNYo

▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
🔥 Optionally, pay us a coffee to boost our Coffee Bean production! ☕
Patreon: https://www.patreon.com/AICoffeeBreak
Ko-fi: https://ko-fi.com/aicoffeebreak
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

Paper links in order of appearance:
* 00:29 ImageNet challenge SOTA -- https://paperswithcode.com/sota/image-classification-on-imagenet
* 00:58 Word2Vec -- Mikolov, Tomas, et al. "Distributed representations of words and phrases and their compositionality." Advances in neural information processing systems. 2013. https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf
* 03:29 The Transformer -- Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems. 2017. https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
* 04:30 Translating Programming Languages -- Lachaux, Marie-Anne, et al. “Unsupervised Translation of Programming Languages”. 2020 https://arxiv.org/pdf/2006.03511.pdf
* 04:32 Symbolic Mathematics -- Lample, Guillaume, and François Charton. "Deep learning for symbolic mathematics." arXiv preprint arXiv:1912.01412 (2019). https://arxiv.org/pdf/1912.01412.pdf
* 04:37 Transformer Demo from Huggingface -- https://transformer.huggingface.co/
* 04:52 BERT -- Devlin, Jacob, et al. "BERT: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018). https://arxiv.org/pdf/1810.04805.pdf
* 06:29 Image Transformer -- Parmar, Niki, et al. "Image transformer." arXiv preprint arXiv:1802.05751 (2018). https://arxiv.org/pdf/1802.05751.pdf

🔗 Links:
YouTube: https://www.youtube.com/channel/UCobqgqE4i5Kf7wrxRxhToQA/
Twitter: https://twitter.com/AICoffeeBreak
Reddit: https://www.reddit.com/r/AICoffeeBreak/

#AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research #TransformerinML

Видео A brief history of the Transformer architecture in NLP канала AI Coffee Break with Letitia