Is chain-of-thought (CoT) a sign of true reasoning in LLMs?

​In this talk our guest, Akshara Prabhakar, an Applied Scientist at Salesforce AI, aims to deepen our understanding of the reasoning process undertaken in Large Language Models (LLMs).

The chain-of-thought (CoT) prompting method can dramatically improve LLM performance by significantly enhancing the multi-step reasoning capabilities. But what type of reasoning do LLMs use when performing CoT — is it genuine reasoning, or is it driven by shallow heuristics like memorization?

In the recently published paper in NAACL, Akshara tackles this question by focusing on the symbolic reasoning task of decoding shift ciphers. Akshara developed a simple probabilistic approach to identify these factors — the probability of the task's expected output (probabilistic effect), what the model has implicitly learned during pre-training (memorization influenced), and the model's attempt to adopt shorter reasoning steps (noisy reasoning) and show the consequent drastic variability in task accuracy.

Through a series of experiments, the conclusion is that LLM behavior exhibits clear hallmarks of both memorization and true reasoning suggesting that CoT resembles a probabilistic, memorization-influenced form of noisy reasoning.

Join the BuzzRobot Slack community to stay in touch.