"Transformer Through the Lens of Predictive Coding Theory"

"Transformer Through the Lens of Predictive Coding Theory"

Is the way modern AI predicts the next word fundamentally similar to how the human brain works?

A new paper explores this fascinating intersection between large language models and neuroscience.

Full PDF available here:
https://www.academia.edu/165341247/THE_TRANSFORMER_THROUGH_THE_LENS_OF_PREDICTIVE_CODING_THEORY

Would love to hear your thoughts 👇

#AI #Neuroscience #PredictiveCoding

Summary
This article examines the Transformer architecture through the lens of predictive coding theory — one of the most influential frameworks in contemporary neuroscience. Predictive coding posits that the brain functions as a hierarchical prediction machine that continuously generates expectations about sensory input and minimizes prediction errors (Rao & Ballard, 1999; Friston, 2010, 2019).
The core mechanism of modern large language models — autoregressive next-token prediction — bears striking similarities to this biological principle. Both systems operate by generating predictions based on prior context and updating internal models to reduce future errors. Recent neuro-AI studies have demonstrated that Transformer layers mirror the hierarchical organization of the human language network, with lower layers capturing local syntactic features and higher layers encoding increasingly abstract semantic and pragmatic representations (Caucheteux & King, 2024; Goldstein et al., 2024).
While significant analogies exist — particularly in hierarchical prediction, attention as precision weighting, and error minimization — important differences remain. Unlike the brain, standard Transformers rely on global backpropagation rather than local error signals, lack active inference, and operate without embodied grounding.
The article discusses promising directions for developing a “Predictive Transformer 2.0,” including explicit prediction-error pathways, oscillatory mechanisms inspired by gamma and theta waves, integration of active inference, multimodal embodied grounding, and local Hebbian-style learning.
Ultimately, viewing the Transformer through predictive coding theory reveals it not merely as a statistical language model, but as one of the most faithful artificial implementations of a fundamental brain mechanism: continuous generation and correction of predictions. This perspective opens new pathways toward AI systems that may one day achieve a deeper, more human-like understanding of the world.

Видео "Transformer Through the Lens of Predictive Coding Theory" канала waldemar wietrzykowski