Cover Image for 9030 club reading "Grokking: Generalization Beyond Overfitting": #62 - ML Paper Reading Group
Cover Image for 9030 club reading "Grokking: Generalization Beyond Overfitting": #62 - ML Paper Reading Group
Avatar for 90/30 Club
Presented by
90/30 Club
We meet weekly in-person to talk about new ML papers! Come and join the discussion!
44 Went

9030 club reading "Grokking: Generalization Beyond Overfitting": #62 - ML Paper Reading Group

Register to See Address
San Francisco, CA
Registration
Past Event
Welcome! To join the event, please register below.
About Event

β€‹β˜•οΈπŸ“ Paper Link πŸ“β˜•οΈ

​https://arxiv.org/pdf/2201.02177

​Abstract; Grokking: Generalization Beyond Overfitting investigates a surprising phenomenon in deep learning where a model first memorizes the training data, appears to overfit for an extended period, and then, after many additional optimization steps, suddenly transitions to near-perfect generalization without any changes to the data or architecture. Through experiments on algorithmic tasks such as modular arithmetic, the authors show that this abrupt shift is driven by the interplay between optimization and implicit regularization, particularly weight decay, rather than by traditional notions of early stopping or explicit changes in training. The work challenges the common assumption that generalization develops gradually alongside decreasing training loss, suggesting instead that neural networks may internally reorganize their representations before exhibiting qualitatively different behavior. More broadly, the paper raises fundamental questions about how models acquire abstract algorithms, why memorization can precede genuine understanding, and whether similar delayed transitions underlie the emergence of reasoning abilities in today's large language models.

​Join us at Mox to explore:

​‒ If a neural network can memorize data for millions of optimization steps before suddenly discovering a more general algorithm, what does this suggest about how intelligence and reasoning emerge during training? Are today's large language models experiencing similar "grokking" phases at scale?

​‒ Grokking demonstrates that optimization can eventually favor simple, generalizable solutions over memorized ones without changing the objective. Does this imply that many capabilities we attribute to model architecture are instead emergent properties of long-term optimization, and how should this influence the way we train future foundation models?

​Event Schedule;

​7pm to 8pm --> quiet reading time, grab and snack and read! (optional)

​8pm to 9pm --> open discussion about reading πŸ“

​9pm --> we have our space for a bit longer, stay to socialize or network!

​Our event is hosted within Mox SF, the gracious donors of the space we will meet.

Location
Please register to see the exact location of this event.
San Francisco, CA
Avatar for 90/30 Club
Presented by
90/30 Club
We meet weekly in-person to talk about new ML papers! Come and join the discussion!
44 Went