90/30 Club (ML reading) #37: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations
Week 37: PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations
The Paper Link Here
PLUM reframes large-scale recommendation as a language modeling problem, replacing massive embedding tables with Semantic IDs (SIDs) and autoregressive generation. Items are discretized via an enhanced RQ-VAE that captures both content semantics and co-occurrence structure, then aligned with user behavior through large-scale continued pre-training. At inference, recommendations are produced by directly generating item IDs rather than retrieving via dot-product similarity, shifting system capacity from sparse embeddings into the model itself. Empirically, PLUM matches or exceeds highly optimized industrial recommender systems at YouTube scale, while scaling cleanly to very large vocabularies and multi-billion-parameter models. The work argues that, with careful representation design and training alignment, generative retrieval can inherit the scaling advantages of LLMs and blur the boundary between language modeling, search, and recommendation.
Join us at Mox to explore:
- How does replacing embedding tables with Semantic IDs change the scaling laws of recommender systems?
- Why is continued pre-training essential for aligning LLMs with user behavior and item structure?
Discussion at 20:00, (optional) quiet reading from 19:00.