Archived2026verified from source

FlakeHunter

Diagnoses a flaky test by experiment, writes a real fix, then proves the fix by running the test 500 times.

placeholder FlakeHunter — Diagnoses a flaky test by experiment, writes a real fix, then proves the fix by running the test 500 times.

Why it exists

A flaky test gets rerun instead of fixed, so real regressions hide behind it. The hard part was never generating a plausible fix, it was confirming one.

How it works

01Hypotheses come with predictions that discriminate between causes, tested by experiments from a closed vocabulary
02An anti-cheat validator rejects patches that only mask the flake
03Verification re-runs the patched test 500 times, including under CPU oversubscription
04A single-call control reported high confidence on all 12 cases, and 2 of those patches did not fix anything

Back to all work