Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy

Paper: Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy (2603.13552)
Published: 13 Mar 2026.

Learn more on Emergent Mind: https://www.emergentmind.com/papers/2603.13552
arXiv: https://arxiv.org/abs/2603.13552
Sign up for our free trending papers email digest: https://www.emergentmind.com/subscribe
Follow us on X: https://x.com/EmergentMind
Join our Discord: https://discord.gg/BhfTC4mTXq

This presentation explores a fundamental geometric constraint governing neural network training with cross-entropy loss. The paper reveals how complex singularities in the softmax partition function—termed 'ghosts of softmax'—create branch points that dictate safe step sizes through Taylor convergence radius, independent of real-line curvature. The authors introduce a tractable bound based on directional logit derivatives that predicts instabilities missed by traditional smoothness-based analysis, and demonstrate a robust controller that survives extreme learning rate perturbations while achieving competitive accuracy without hand-tuned schedules.

Видео Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy канала Emergent Mind