S3-E5 · Is Your Million-Token AI Lying? (How Context Really Works)

A model advertises a one million token context window, roughly a thousand pages held in mind at once, so you feed it the whole contract or the full codebase and ask one careful question, and it quietly misses a fact buried in the middle. This lecture explains the gap between the advertised window and the effective one, without the lazy 'AI can't do long context' framing, because 2026 already moved. You will understand RoPE (how position is baked in as a rotation so attention reads relative distance from an angle), how that rotation is stretched past its training length with tricks like YaRN, why sliding-window attention trades cost for reach, and where hybrid Mamba and state-space models fit. We lead with the labs' own numbers so you can judge a long-context claim honestly.

Full course playlist: https://www.youtube.com/playlist?list=PLY1vE5Urmcu8

New lecture every week. Subscribe to @HowAIWorksHQ to understand how AI really works, one clear idea at a time.

Видео S3-E5 · Is Your Million-Token AI Lying? (How Context Really Works) канала How AI Works?