Cover Image for Simulating and evaluating spoken conversations
Cover Image for Simulating and evaluating spoken conversations
Avatar for NAVER LABS Europe seminars

Simulating and evaluating spoken conversations

Zoom
Registration
Past Event
Welcome! To join the event, please register below.
About Event

​NAVER LABS Europe seminars are open to the public. Please register to participate.

Abstract: Since the release of Moshi in 2024, full-duplex models have become a popular choice for spoken conversational systems thanks to their ability to listen and speak at the same time, allowing them to handle backchannels and interruptions smoothly. However, evaluation of these capabilities has been limited because it has relied either on static prerecorded conversations, which cannot react to the tested system, or on cascaded turn-based simulators, which typically do not produce user behaviours such as backchannels or interruptions. This motivated our JSALT 2026 project to build a full-duplex user simulator capable of testing spoken conversational systems in dynamic simulated conversations.
I will describe our work on controlling the simulator’s conversational behaviours and content using constrained decoding algorithms and collaboration with an external text LLM. I will then present our approaches to evaluating and ranking spoken conversational systems by comparing the distributions of relevant features, such as speaking rate or turn durations, extracted from simulated conversations with those extracted from real conversations.

​Ondrej Klejch is a Senior Researcher in the Centre for Speech Technology Research, School of Informatics, University of Edinburgh. His work focuses on building automatic speech recognition systems with limited training data and supervision. More recently, he has been working on spoken conversational systems. He led the JSALT 2026 project on simulating and evaluating spoken conversations and is one of the organisers of the ICASSP 2027 workshop, EVA: Evaluation of Voice Agents.

Avatar for NAVER LABS Europe seminars