The Model Is the Easy Part: Shipping an LLM Judge for Systematic Reviews

​Every systematic review starts with a brutal chore: someone has to read thousands of scientific abstracts and decide, one by one, whether each paper meets the inclusion criteria. It takes weeks, it is mind-numbing, and it is exactly the kind of structured judgement an LLM is good at, provided you can trust it.

​This talk is the story of PaperJudge, a web application that turns that chore into a pipeline. We start with the problem and how it grew into a product with real users and real invoices. Then we open the hood: a Kotlin and Spring Boot modular monolith built on Spring Modulith, a composable Pipeline pattern for multi-step workflows, sealed Result types instead of exceptions, Java 21 virtual threads to classify articles in parallel, schema-per-tenant PostgreSQL isolation, Stripe metered billing, and a React frontend.

​The final part is about the judge itself, and how it changed. Version one ran GPT-4.1 at temperature 0: one deterministic pass per abstract, returning a structured include, exclude, or uncertain decision with a confidence score and a justification anchored to the inclusion criteria. Then we moved to gpt-5.6-luna, cheaper and smarter, but that took determinism off the table. So we replaced the single deterministic pass with three independent run cycles per abstract, and let the agreement between runs, rather than a fixed temperature, decide what the judge says and how much it trusts itself. We will show what that did to accuracy, cost, and the "uncertain" pile, and the failure modes we hit along the way.

​You will leave with a concrete blueprint for putting an LLM judge behind a real product, and a list of mistakes you can skip.