Anatoly Potapov: Pre-training Transformers with Catalyst

Data fest Online 2020
Catalyst Workshop track https://ods.ai/tracks/catalyst-df2020

Since the NLP's Imagenet moment (emergence of ELMO, BERT), language models are widely used as a backbone for a variety of supervised tasks (intent classification, named entity recognition, question answering), etc

At Tinkoff, we have tens of millions of unlabelled customer conversation samples. In such a scenario, it is highly beneficial to pre-train on in-domain data and having a custom vocabulary. We transferred our pre-training pipeline to the Catalyst framework and reduced our codebase while keeping features like distributed training and fp16 training.

In my presentation, I will tell you how to pre-train transformers at scale with the Catalyst framework without writing lots of "infrastructure" code.

Register and get access to the tracks: https://ods.ai/events/datafest2020
Join the community: https://ods.ai/

Видео Anatoly Potapov: Pre-training Transformers with Catalyst канала ODS AI Global