Arvind Neelakantan: Text and Code Embeddings
Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, Arvind Neelakantan (OpenAI) shows that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. Similarly to text embeddings, he trains code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.
Видео Arvind Neelakantan: Text and Code Embeddings канала Oxford ML and Physics Seminars
Видео Arvind Neelakantan: Text and Code Embeddings канала Oxford ML and Physics Seminars
Показать
Комментарии отсутствуют
Информация о видео
13 июня 2022 г. 14:00:45
00:36:36
Другие видео канала
![Mike Walmsley: Galaxy Zoo(m): Probabilistic Galaxy Morphology via Bayesian CNNs and Active Learning](https://i.ytimg.com/vi/RNiZMrjH9EI/default.jpg)
![Rachel Prudden: Probabilistic modelling for atmospheric science: beyond the noise](https://i.ytimg.com/vi/hRVqjOapj94/default.jpg)
![Ricardo Vinuesa: Artificial Intelligence, Computational Fluid Dynamics, and Sustainability](https://i.ytimg.com/vi/afO8e7knqD8/default.jpg)
![Atılım Güneş Baydin: Probabilistic Programming for Inverse Problems in the Physical Sciences](https://i.ytimg.com/vi/E3_Ey0z068o/default.jpg)
![Ard Louis: Deep neural networks have an inbuilt Occam’s razor](https://i.ytimg.com/vi/tzXhbZhNLoU/default.jpg)
![Guillaume Lample: Deep Learning for Symbolic Mathematics](https://i.ytimg.com/vi/jMx_68VxKiA/default.jpg)
![Brian Spears: Cognitive Simulation: combining simulation and experiment with artificial intelligence](https://i.ytimg.com/vi/oHMhxqkDegg/default.jpg)
![Eliu Huerta: AI for Science: Let’s talk business](https://i.ytimg.com/vi/lgmROvsz9WI/default.jpg)
![Ben Nachman: Extracting the most from collider data with deep learning](https://i.ytimg.com/vi/nrhu2rMK28c/default.jpg)
![Laure Zanna: Climate Modeling in the Age of Machine Learning](https://i.ytimg.com/vi/roOx01QmS20/default.jpg)
![Tim Green: Highly accurate protein structure prediction with AlphaFold](https://i.ytimg.com/vi/PqK1Oi9Pr10/default.jpg)
![Peter Dueben: Machine learning for weather predictions](https://i.ytimg.com/vi/V2JAaD-qx4U/default.jpg)
![David Spergel: Determining the Universe’s Initial Conditions](https://i.ytimg.com/vi/xYqN5dZjIQY/default.jpg)
![Adrien Gaidon: Self-supervised 3D vision](https://i.ytimg.com/vi/3YZNiM2l8Z0/default.jpg)
![Maurizio Pierini: Doing more with less: Deep Learning for Physics at the Large Hadron Collider](https://i.ytimg.com/vi/tzsNkhkaexE/default.jpg)
![Michael Kagan: Generative Model Based Design Optimization and Unfolding](https://i.ytimg.com/vi/Ytlb6ux9wMU/default.jpg)
![Séamus Davis: Machine learning in electronic-quantum-matter imaging experiments](https://i.ytimg.com/vi/hVwWQF2envw/default.jpg)
![Stéphane Mallat: Hamiltonian Estimations by Conditional Renormalisation Group and Convolution Nets](https://i.ytimg.com/vi/3LvWKwGQXPk/default.jpg)
![Phiala Shanahan: Provably exact sampling for first-principles theoretical physics](https://i.ytimg.com/vi/J2zpdmh0EOI/default.jpg)
![Jonas Buchli & Federico Felici: Magnetic control of tokamak plasmas with deep reinforcement learning](https://i.ytimg.com/vi/Oesjkp5tSkA/default.jpg)