Integer embeddings in PyTorch

In this video, we implement a paper called "Learning Mathematical Properties of Integers". Most notably, we use an LSTM network and an Encyclopedia of integer sequences to train custom integer embeddings. At the same time, we also extract integer sequences from already pretrained models - BERT and GloVe. We then compare how good these embeddings are at encoding mathematical properties of integers (like divisibility by 2 and primality).
Paper: https://arxiv.org/abs/2109.07230
Code from this video: https://github.com/jankrepl/mildlyoverfitted/tree/master/github_adventures/integer

00:00 Intro
00:41 Ideas and high level explanation
02:56 Data - On-line encyclopedia of Integer Sequences
03:58 Data - raw download exploration
05:43 CustomDataset - implementation
09:15 CustomDataset - testing it out
11:36 Network - implementation
15:54 Network - testing it out
16:58 Evaluation utilities
19:05 GloVe embeddings parsing
22:09 BERT embeddings parsing
24:32 LSTM training script
30:58 Experiments to be run
31:39 Results: LSTM guess next
33:53 Results: Metrics (TensorBoard)
36:33 Results: Embeddings projections (TensorBoard)
40:05 Outro

Видео Integer embeddings in PyTorch канала mildlyoverfitted