The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding

Paper: https://arxiv.org/pdf/2502.08946
NotebookLM(Request Access): https://notebooklm.google.com/notebook/10f791e1-7296-4c2a-bb3b-be7b579b2427?original_referer=https:%2F%2Fwww.google.com%23&pli=1

This research addresses the question of whether large language models (LLMs) really understand physical concepts, or whether they are simply “stochastic parrots” repeating information. To investigate this, the authors introduce PHYSICO, a new dataset that assesses the comprehension of physical concepts through abstract inputs in grid format. The results show that LLMs, even the most advanced LLMs such as GPT-4o, perform significantly worse than humans on high-level comprehension tasks, while performing well on natural language recognition of physical concepts. This supports the idea that LLMs exhibit the “stochastic parrot” phenomenon, lacking true deep comprehension. Furthermore, experiments with context learning and fine-tuning did not substantially improve LLMs' performance in PHYSICO, suggesting that the problem lies in LLMs' intrinsic limitations in conceptual understanding.

Видео The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding канала AI Papers Decoded Podcast