Frontier Paper Club
Frontier Paper Club — NYC
Papers. People. Possibilities.
How do we know an AI agent is actually getting better? What makes a research result hold up beyond its benchmark? And which ideas are ready to become systems people can depend on?
Frontier Paper Club is a monthly gathering in New York for researchers, engineers, and founders exploring the next generation of AI. Sponsored by Betaworks and Blobfish.ai, we bring together people developing new methods, building agent systems, and testing what works in practice.
Each session centers on 2–3 papers, benchmarks, or technical implementations selected around a shared research question. Topics span AI agents, reinforcement learning, post-training, reasoning, memory, evaluation, and simulation environments.
Presentations are brief and technically grounded, leaving plenty of time to examine the work together: the assumptions behind a method, the strength of its evidence, the details needed to reproduce it, and the failures that suggest where research should go next.
🕒 Evening format
5:30pm Arrivals & Introductions
Meet fellow researchers and builders. Share what you’re working on and the questions you’re hoping to explore.
6:00pm Research Presentations & Benchmark Discussion
Presenters walk through the problem, approach, key results, and limitations. Each talk is followed by audience questions and technical discussion.
8:00pm Open Conversation & Networking
Continue the discussion, exchange implementation notes, and meet potential collaborators.
The detailed schedule will be announced with the speaker lineup.
Random Remasking Scales Masked Diffusion Inference - https://arxiv.org/pdf/2610.30744
Masked diffusion models (MDMs) generate sequences by progressively unmasking tokens, but errors introduced during decoding are typically irreversible. Recent work has shown that iterative correction can improve sample quality, yet existing theory fails to explain this given a learned, imperfect denoiser model. We re-formulate MDM inference as a Markov chain on the clean sequence space, defined by a remask-denoise kernel that randomly re-introduces masks and then denoises. This helps us analyze the actual denoising trajectory towards the data distribution. The analysis motivates Random Remask-Denoise (R^2D) inference, a purely test-time, finetuning-free, reward-free procedure. To further improve performance, we propose Trajectory Correction Training (TCT), a denoiser-training procedure that exposes the model to its own errors along the generation trajectory. On logical tasks, our approach exhibits a striking near-log-linear test-time scaling, achieving over 97% accuracy on hard Sudoku puzzles and over 99% accuracy on 3-SAT. Our approach also applies to pretrained models on actual language tasks, beating iterative correction baselines and the official Q-mode accuracy of LLaDA2.1-mini on HumanEval+.
When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers - https://arxiv.org/abs/2512.02304
Large language models (LLMs) can act as both problem solvers and solution verifiers, where the latter select high-quality answers from a pool of solver-generated candidates. This raises the question of under what conditions verification pays off in solver-verifier systems. Prior work has conducted only limited studies of the factors influencing verification performance, focusing primarily on self-verification and examining neither the relationship between solver and verifier model families nor the effects of reasoning post-training. To rectify this, we present a systematic study across 37 models spanning multiple families, sizes, and base vs. post-trained variants, evaluated on 9 benchmarks covering logical reasoning, structured puzzles, symbolic computation, mathematics, commonsense, factual recall, and domain knowledge. In order to support our analysis, we introduce and empirically validate verifier gain, a metric that predicts the performance improvements from test-time verifier-based rejection sampling. Our experiments find that 1) verification across model families is more effective than either self-verification or verification within the same family, and more generally that the benefits of verification decrease as the solver and verifier become more similar, 2) reasoning post-training weakens self-improvement abilities but strengthens cross-family improvement, and 3) some tasks are inherently more amenable to improvement through verification, particularly mathematical and logical tasks.
AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance - https://arxiv.org/abs/2506.03828
AI for Industrial Asset Lifecycle Management aims to automate complex operational workflows, such as condition monitoring and maintenance scheduling, to minimize system downtime. While traditional AI/ML approaches solve narrow tasks in isolation, Large Language Model (LLM) agents offer a next-generation opportunity for end-to-end automation. In this paper, we introduce AssetOpsBench, a unified framework for orchestrating and evaluating domain-specific agents for Industry 4.0. AssetOpsBench provides a multimodal ecosystem comprising a catalog of four domain-specific agents, a curated dataset of 140+ human-authored natural-language queries grounded in real industrial scenarios, and a simulated, CouchDB-backed IoT environment. We introduce an automated evaluation framework that uses three key metrics to analyze architectural trade-offs between the Tool-As-Agent and Plan-Executor paradigms, along with a systematic procedure for the automated discovery of emerging failure modes. The practical relevance of AssetOpsBench is demonstrated by its broad community adoption, with 250+ users and over 500 agents submitted to our public benchmarking platform, supporting reproducible and scalable research for real-world industrial operations. (2.4k stars IBM/AssetOpsBench)
📝 Want to present your work?
Have research or an implementation you’d like to discuss with New York’s AI community?
We’re accepting submissions covering:
AI agents, reasoning, planning, and memory
Reinforcement learning and post-training
Benchmarks, evaluation methods, and agent reliability
Simulation environments and synthetic training data
Reproductions, negative results, and practical lessons from deployment
Share a link to your paper, benchmark, project, or slides, along with what you’d like to present. Submissions will be considered for upcoming sessions.
Questions? Contact sam@blobfish.ai.
🤝 Sponsored by
Betaworks builds and invests in technology companies, with a longstanding presence in New York’s startup community and a focus on emerging technologies, including AI, agents, and developer tools. Through its thematic investment and product development program, Camp, it brings founders together to explore new categories and build companies. Its history includes backing companies such as Hugging Face and Stability AI.
Blobfish.ai builds training gyms for AI agents—simulated enterprise environments where models practice workflows across tools such as CRMs, ERPs, and spreadsheets.
Its environments combine realistic data, executable tools, adjustable task difficulty, and deterministic checks of task outcomes. As models improve, Blobfish generates new challenges to support continued learning, with a focus on the complex, multi-step work agents need to perform in practice.