Learning Layer Paper Reading Club - Week 35 - Noisy MoE Routing for Efficient Search and Dynamic Compute Allocation
This week's paper:
Noisy MoE Routing for Efficient Search
Link:
https://voaige.com/research/noisy-moe-routing-for-efficient-search
Presenter:
This week's discussion lead is Max Ladow, a founder of Voaige voaige. Voaige is an AI research lab bringing neuroscientists, AI researchers, and engineers together to build System 2 AI through neuroscience. We translate findings from cognitive and systems neuroscience into computational principles for intelligence. We then engineer and test them in language models under the demands of real-world performance and cost.
Abstract:
We investigate whether perturbing expert routing improves inference-time search in mixture-of-experts language models. Our training-free method adds Gaussian noise to router weights, holding perturbations fixed within each reasoning trajectory and applying them to a small fraction of routing decisions. On mathematical reasoning tasks with GPT-OSS-120B, intervening in approximately 1–2% of decisions improves hard-problem Pass@32 by 3.0 percentage points, without observed degradation on easier problems. Higher intervention rates generally reduce performance, while resampling perturbations at each decision removes the benefit on the hardest problems. These preliminary findings suggest that coherent variation in expert routing can expose useful reasoning trajectories, although generalization beyond the evaluated model and domain remains untested.
Anyone curious about AI research is welcome, whatever your background. No preparation required — come read and discuss with us.
What is the Learning Layer Labs Paper Reading Club?
An initiative from https://www.learninglayer.ai, a lab with the goal of reducing AI anxiety in the world.
What is the format?
Discussion based. Expect a low pressure environment to share insights and opinions with the group.
What are the group goals?
Stay on top of AI research and improve understanding of AI fundamentals + math.
Who is welcome?
Everyone! Try to put in at least some time on the paper and come prepared with questions or things you'd like to discuss, but it's ok to just show up!
Learning Layer Labs team:
Thomas Redfern ( The backbone who runs every paper reading )
Mat Allen (The new fella bring order and organization)
Devinder Sodhi (The guy you talk to for sponsoring)