Flake it till you make it: how to detect and deal with flaky tests (Ep. 528)
On today’s episode we chat with Prof. Gregory Kapfhammer, Associate Professor in the Department of Computer Science at Allegheny College, and a specialist in the subject of flaky tests. He breaks down his definition of what makes a test flaky, how you can solve for it, and automated solutions that take some of the guesswork and busywork out of avoiding them altogether.
There is a ton of great research to be found on Prof. Kapfhammer’s website, including:
- Flaky Tests: Finding and fixing unpredictable and harmful test cases
- Database Testing: Automatically testing relational database schemas
- Web Testing: Detecting and repairing poor responsive web page layout
We’ve written a bit about how Stack Overflow is upping its unit testing game and how you can evaluate multiple assertions in a single test.
Thanks to our lifeboat badge winner of the week, Survivor, for answering the question: Is it possible to find out if a value exists twice in an arraylist?
And thanks to Andre for writing in to let us know that yes, there are folks who code on mobile. He suggests trying:
– Termux https://wiki.termux.com/wiki/Main_Page can be used as a
terminal emulator. I have all I need here: Vim, git, node.js, Python
– Markor https://gsantner.net/project/markor.html has a HTML view
React has a lot of things to be aware, since unit tests may be impacted by component of global states.
States may change inside a test, and often, state change could finish after test finishes. In that case your test will be wrong and could be flaky, in case the state change finish before test finishes.
Computers are deterministic. When given the same input, you will always get the same output. If you’re not getting the same output, you do not have the same input.
All flakiness is due to uncontrolled inputs. If a test is waiting on a poorly understood asynchronous process, the test may fail when the process does not complete in the expected time. The asynchronous process is an uncontrolled input.
The answer to all flakiness is to understand all test variables and control them. The answer is that simple. The implementation is left as an exercise to the reader.
That was a great listen, thanks for producing this episode! 🙂 The transcription spells Gregory’s phd pal’s name wrong, so if you’re looking for their research on flaky tests, the spelling is Owain. Everywhere I work tests are a big problem, so it was really nice to discover all this fresh research!