Language Model Overview: From word2vec to BERT

Language Model Overview, presented in ServiceNow
Covered list:
A Neural Probabilistic Language Model (NNML) http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf
Efficient Estimation of Word Representations in Vector Space https://arxiv.org/pdf/1301.3781.pdf
Distributed Representations of Words and Phrases and their Compositionality https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf
GloVe: Global Vectors for Word Representation https://nlp.stanford.edu/pubs/glove.pdf
Distributed Representations of Sentences and Documents https://cs.stanford.edu/~quocle/paragraph_vector.pdf
Skip-Thought Vectors https://arxiv.org/pdf/1506.06726.pdf https://github.com/ryankiros/skip-thoughts
AN EFFICIENT FRAMEWORK FOR LEARNING SENTENCE REPRESENTATIONS https://arxiv.org/pdf/1803.02893.pdf https://github.com/lajanugen/S2V
Supervised Learning of Universal Sentence Representations from Natural Language Inference Datahttps://arxiv.org/pdf/1705.02364.pdf https://github.com/facebookresearch/InferSent https://github.com/facebookresearch/SentEval.git
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning https://arxiv.org/abs/1804.00079 https://github.com/Maluuba/gensen
Universal Sentence Encoder https://arxiv.org/abs/1803.11175 https://tfhub.dev/google/universal-sentence-encoder/1 https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb#scrollTo=eAVQGidpL8v5
Learned in Translation: Contextualized Word Vectors https://arxiv.org/pdf/1708.00107.pdf https://github.com/salesforce/cove
Deep contextualized word representations http://www.aclweb.org/anthology/N18-1202 https://allennlp.org/elmo https://github.com/allenai/bilm-tf
Universal Language Model Fine-tuning for Text Classification (ULMFit) https://arxiv.org/abs/1801.06146 https://github.com/akzaidi/fine-lm
Improving Language Understanding by Generative Pre-Training (GPT) https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf https://github.com/openai/finetune-transformer-lm
Attention Is All You Need https://github.com/jadore801120/attention-is-all-you-need-pytorch https://arxiv.org/pdf/1706.03762.pdf
Generating Wikipedia by Summarizing Long Sequences https://arxiv.org/abs/1801.10198 https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/data_generators/wikisum
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding https://arxiv.org/abs/1810.04805 https://github.com/google-research/bert

Видео Language Model Overview: From word2vec to BERT канала James King