Would your app hold at 240 users? A load test found its production autoscaling could not scale
For CTOs and heads of platform at companies running customer apps on Kubernetes.
We ran a k6 load test of a resident mobile app on a staging cluster. Each fix exposed the next bottleneck, and the chain ended at something no functional test could find.
- Property management software
- Resident mobile app on Kubernetes
- Clients
- Money
- Security
What was at risk
Residents waiting on slow screens
The p95 went from 745 ms at 30 users to 2.6 s at 60, with zero errors, so no error monitor would have said anything.
Autoscalers that could not scale
On the day of the test, production had autoscalers on the busiest services, but the cluster had no metrics server, so every target read unknown and none could scale.
A rate limit anyone could skip
The services took the client address from a caller-controlled forwarded header, so a new value on every request skipped every per-IP limit. It was fixed during the engagement.
Standards and rules in play
- A safe-by-design load scriptRead-only: ten GET requests per session with four to nine seconds of think time, so it leaves no data behind. Large runs went inside the cluster, with a forwarded address per virtual user.
Context: the rule in play is your own availability target.
How we tested it
- Test level
- System testing, non-functional, against a staging cluster.
- Test types
- Performance testing, specifically load testing, with a security observation on the way.
- Test basis
- A resident's typical session: home, payments, services and community.
- How we workedLoad testing by iterationWhere it showed upRun, clear one bottleneck, run again: CPU limits first, then one permission service, then its replicas.
- How we workedConfiguration reviewWhere it showed upThe missing metrics server and the nodes that could not grow were found by reading the cluster, not by any functional test.
Terms as in the ISTQB CTFL 4.0.1 syllabus and glossary.
What we found, and what it costs
- FindingFound745 ms at 30 users, 2.6 s at 60, with zero errorsWhat it costs youSlowness without errors never shows in an error monitor. Two of the busiest services were at their CPU limit.
- FindingFoundWith limits raised, one permission service became the limit: p95 of 10 to 16 s at 240 usersWhat it costs youFix one bottleneck and the next appears. The database and the cache were never the constraint.
- FindingCritical pathProduction autoscalers could not scale: no metrics serverWhat it costs youThe safety net for a traffic spike was not there. This is the state we found on the day of the test.
- FindingSecurityA caller-controlled forwarded address skipped every per-IP rate limitWhat it costs youFixed during the engagement.
Once autoscaling worked in staging, new pods failed to schedule because the nodes were nearly full and nothing added nodes. Requests from inside the network share one empty address, and so one rate budget.
What changed
Two replicas of the permission service and a full core each took 240 users to about 200 requests a second at a p95 of 169 ms, with zero errors in any run. The staging cluster now has a metrics server, and the load script can be rerun.
Will your app hold when real traffic arrives?
We load test inside the cluster and follow the chain of bottlenecks to the end. The first month is a trial.
