AI Journal Club for Researchers ft. Ben Coleman (Google Deepmind)
Join the Workato AI Research Lab for small, discussion-driven sessions with fellow AI researchers.
These gatherings will bring together researchers to share recent papers, discuss ongoing work, and exchange perspectives on how AI research is shaping real world systems. The focus is on open dialogue, technical depth, and learning from peers working at the forefront of their field.
Featured Speaker
Ben Coleman, Research Scientist at Google Deepmind
Benjamin Coleman is a senior research scientist at Google DeepMind. He received his PhD from Rice University advised by Anshumali Shrivastava. Ben's research seeks to improve the Pareto frontier for the tradeoffs between ML model performance, data quantity/quantity, and cost. In past years, he studied this problem using techniques from information theory, sampling, and randomized algorithms. More recently, his work focuses on the development of research agents that improve the tradeoffs through the automated discovery of new ML methods.
Context Management for Complex Agentic Workflows
LLMs have rapidly transitioned from simple dialogue machines into capable agentic systems that execute sequences of actions to pursue complex goals. This shift has enabled the application of LLMs to a much wider set of problems, and LLM agents achieve excellent performance on tasks that can be solved with short sequences of actions. However, LLMs often struggle with tasks that are long-horizon, high-context, and require learning from experience. This limits their ability to repeatedly interact with the environment to solve a task, posing fundamental challenges for LLM-driven scientific experimentation, programming, and research.
In this talk, we investigate the reasons why agents fail on complex tasks and propose context management solutions for important failure modes. For example, agents often fail to distill a long experience log into reusable procedural knowledge. We address this problem in our Evo-Memory paper, where we benchmark the agent's ability to learn from experience. To do this, we rephrase existing evals as a stream of tasks where the LLM is allowed to maintain a persistent state between each task. To manipulate and evolve this state, we introduce LLM-driven retrospection techniques (ExpRAG and ReMem), ultimately finding that iterative memory refinement is a critical feature for procedural learning.
Next, we turn our attention to evolution agents for hill-climbing problems, where the LLM is asked to iterate on a design based on design quality feedback. In this setting, we find that LLMs suffer from pathological context-conditioning effects: they often propose ideas similar to those already in context. This prevents adequate exploration of the solution space and results in a negative feedback loop where each idea strongly conditions the generation of similar follow-up ideas. We develop an evolution framework (PACEvolve) that breaks this loop with context-sampling mechanisms to enable backtracking and hierarchical memory management. By addressing the underlying context-conditioning problem, PACEvolve attains SOTA results on benchmarks such as NanoGPT and KernelBench.
Finally, we briefly discuss the promise and limitations of context management as a general tool to fix behavioral problems in LLM agents.
Who Should Attend
AI Researchers and practitioners working at the intersection of AI research and real world systems
About Workato
Workato is an enterprise automation and integration platform that orchestrates workflows across applications, data, and systems enabling secure, governed execution of complex processes as organizations adopt AI agents at scale. You can explore Workato's end-to-end capabilities here.