Paper Reading Night - Week 25 - On the Theoretical Limitations of Embedding-Based Retrieval

​Weekly paper:

​On the Theoretical Limitations of Embedding-Based Retrieval

​https://arxiv.org/html/2508.21038v1

​

​What are the group goals? Stay on top of AI research, improve understanding of AI fundamentals+math.

​Who is welcome? Everyone! Try to put in at least some time on the paper and come prepared with questions or things you'd like to discuss, but it's ok to just show up.

​This paper reading club is brought to you by Learning Layer Labs ( https://www.learninglayer.ai ) and amazing humans like Thomas Redfern + Mat Allen who all pull it together at the AI Floor of Frontier Tower.

We have a sponsor this week! HydraDB (https://www.hydradb.com/) will be sponsor this event and bringing on Aditya Shrivastava to lead the discussion.

​

​Abstract:

​Vector embeddings have been tasked with an ever-increasing set of retrieval tasks over the years, with a nascent rise in using them for reasoning, instruction-following, coding, and more. These new benchmarks push embeddings to work for any query and any notion of relevance that could be given. While prior works have pointed out theoretical limitations of vector embeddings, there is a common assumption that these difficulties are exclusively due to unrealistic queries, and those that are not can be overcome with better training data and larger models. In this work, we demonstrate that we may encounter these theoretical limitations in realistic settings with extremely simple queries. We connect known results in learning theory, showing that the number of top-k subsets of documents capable of being returned as the result of some query is limited by the dimension of the embedding. We empirically show that this holds true even if we restrict to k=2, and directly optimize on the test set with free parameterized embeddings. We then create a realistic dataset called LIMIT that stress tests models based on these theoretical results, and observe that even state-of-the-art models fail on this dataset despite the simple nature of the task. Our work shows the limits of embedding models under the existing single vector paradigm and calls for future research to develop methods that can resolve this fundamental limitation.

​

​This event is hosted at the Frontier Tower:

​We are transforming a 16-floor tower in San Francisco into a self-governed vertical village—a hub for frontier technologies and creative arts. Tier-one labs presenting AI, Ethereum, biotech, neuroscience, longevity, robotics, makerspace, human flourishing, and arts & music. These floors will house innovators and creators pushing the boundaries of human potential in a post-AI-singularity world.

​Apply here for founding citizenship: https://frontiertower.io/apply

​Why should I become a citizen?

• ​Be part of creating the first self-governed vertical village

• ​Connect with the most creative people in the city

• ​Get access to all floors, free event space & movement floor

​Website: https://frontiertower.io/

​Need more reading? Visit https://frontiertower.notion.site/