Breaking Things Before Production Does
Anatomy of an impossible bug: Unleashing deterministic simulation and surviving metastable failures.
Talk 1: Pick your Circle: Suffering for Your Sins at Machine Speed
So you put a proxy in front of MySQL or Postgres, and it uses CDC to incrementally maintain partial result sets so you can use it as a cache without application code changes. What could go wrong? Luckily, SQL and caching are both simple problem domains with straightforward semantics, so this is easy to test and verify. Just kidding: testing SQL is hell, and in this talk Dante will guide us through the 9 circles of the database inferno, focusing on 2 real Readyset bugs found via absurdly overengineered testing methods. Yes, there will be multiple coinciding race windows. Yes, there will be deterministic simulation testing (Antithesis), a Prolog-inspired query generator (Dante), cache invalidation, and off-by-one errors. And yes, the Ninth Circle, reserved for those guilty of treachery and betrayal, is rife with distributed consensus protocols.
Talk 2: Analysis for better resilience
Distributed systems have complicated and unpredictable behaviors, driven by the dynamics of the arriving workload, faults, failures, and interactions with other systems. One of the challenges for operators is to ensure that their system is resilient, meaning that it can tolerate inevitable stressors like overload or hardware failure, and either continue operating, perhaps in a degraded state, or fail gracefully. We have found that simple models and analytical techniques help to ensure the desired level of resilience by demystifying the behavioral dynamics and the trade-offs of complex distributed systems. In this talk I will describe work we’ve done at AWS over the last couple of years to understand the vulnerability of services to metastable failure (congestive collapse), and to reason about the tradeoff between server protection and client availability in retry policies.
About Speakers
Michael Victor Zink is a software engineer at Readyset who got better grades in philosophy than computer science, but liked Paxos and got stuck doing database stuff. At Readyset, he balances shipping feature work with an incessant drive to convert his philosophical derangement ("don't answer questions, question answers") into test infra and verification efforts. Readyset (derived from the Noria research project at MIT) is a database scalability platform featuring a drop-in wire compatible caching layer for Postgres and MySQL.
Rebecca is a senior principal scientist at AWS, where she is part of the DynamoDB organisation, working on resilience and performance. Prior to AWS, she worked at Twitter and Google, largely focused on all aspects of distributed tracing, from trace production through to novel uses of aggregate trace analysis. This followed over a decade at Microsoft Research doing research broadly in the area of performance analysis of distributed and concurrent systems.