Designing Scalable and Rigorous Evaluations for AI Systems
Building an AI application is just the first step. Equally important is knowing whether it performs as expected, produces reliable results, and can be used safely in real-world settings.
This hands-on technical workshop introduces METIS, a specifications-driven approach to functionality evaluations that brings together best practices into a guided testing workflow, including assessing functional performance within agentic traces.
Participants will explore practical approaches to evaluating Large Language Model (LLM) applications and developing more structured evaluations, including:
Identifying what to test
Developing relevant test cases
Setting evaluation criteria
Exploring techniques to augment and strengthen evaluations
Understanding different types of evaluations, from general benchmarks to application-specific tests
Recognising common evaluation challenges and practical ways to address them
Through the workshop, participants will develop a more systematic approach to assessing the quality, reliability and safety of AI applications, and how these approaches can be applied to their own work.
Who Should Attend
This workshop is suitable for Singapore Government officers involved in developing, managing or assessing AI applications, including product managers, engineers, data scientists, policy officers and UX researchers.
It is particularly useful for those looking to adopt a more systematic approach to evaluating AI systems.
No advanced machine learning knowledge or prior experience with AI evaluation is required.
Prerequisites
Participants should have a basic understanding of AI applications, such as LLMs or chatbots, or some experience working with AI-enabled products.
The workshop is designed to help participants understand what to evaluate, how to develop useful test cases, and how to assess whether an AI system is performing as intended.
Important Notes
Please bring a laptop for this hands-on session.
Registration begins at 2.00pm, with the workshop starting at 2.30pm.
This workshop is strictly open to Singapore Government officers only.
Please register using your government email address.
This workshop is part of GovTech's AI Practice division's efforts to uplift Whole of Government capabilities. Read more about how GovTech is building AI for public good.
Get involved: Learn more about Lorong AI | Speaker Sign-up | WhatsApp Community | LinkedIn | X