Nearly half our test runs failed because the database was across the Atlantic
The slow machines were not busy. Each one was slow by the same amount, every time.
Our CI for BetterFlow kept refusing to run the test suite on branches that were not broken. The tell was in the variance: slow machines were not noisy, they were constant.
A red check with nothing wrong
Our CI measures a database round trip before it runs the suite, and refuses to start if a query is far too slow, because at that speed the suite would take a day. Nearly half the runs were being refused, and the branches were fine.
Constant, not noisy
Across 38 runs, every container sat in one of two groups and never moved between them. Fast ones answered in 0.65 to 3.29 ms. Slow ones answered in 136.85 to 147.23 ms. One container returned about 137.7 ms on six consecutive runs, and another returned exactly 136.85 twice.
Load is noisy. A fixed cost that follows a machine is a place, not a busy day.
Two wrong turns first
An early note blamed the network path with a control that proved nothing about it, and it was withdrawn the next day. Later, a comment cleared the database as the cause after measuring the wrong database, a different project's, which was right by luck.
Both were caught by re-reading what had actually been measured. Neither conclusion was used to change anything.
The cause
The runner fleet was set to run 4 replicas in California and 6 in the Netherlands, and the CI database was in California. Every Dutch container paid a transatlantic round trip on every query. Adding replicas could not help, since each new one is another roll of the same dice, and neither could a restart.
We put all ten runners in the database's region. The next two runs passed the gate.
What to take from it
- Read the variance, not the average. No noise means a fixed cost.
- Check what you measured. We measured the wrong database once.
- Make the gate say where. Each run now prints the region next to the timing.
Want to know if your CI is failing for reasons that are not your code?
We measure the pipeline before we blame the branch. The first month is a trial.
