A bad request came back as 'success', in four subsystems of a wealth platform
For CTOs and heads of engineering at fintech and wealth platforms that expose an API to clients or partners.
If partners or clients call your API, some bad requests may be answered as if they worked. While testing a wealth-management API we found the same defect in one subsystem after another: a bad request got HTTP 200, a success flag and a leaked internal error name.
- Wealth management
- Portfolio and mandate API
- Clients
- Operations
- Reputation
What was at risk
Bad requests that look accepted
A bad request got HTTP 200 and a success flag. A client that trusts the flag carries on with nothing, or with a wrong value.
Failures no monitor sees
A status-code monitor sees a healthy call. The failure shows up later in a client's data, not in your alerts.
Internal detail in the answer
The reply carried the name of an internal exception class where the value should be. Callers learn about your internals instead of what they got wrong.
Standards and rules in play
- Our reporting rule for known bugsA check for an open bug stays red and is reported but does not fail the run, and turns green the day the fix lands. A suite that cannot run is reported as not reported, never as a pass.
Context: financial firms in the EU answer to DORA for managing ICT incidents, and a failure that reports success is one nobody can detect.
How we tested it
- Test level
- System testing at API level, run against the live service.
- Test types
- Functional negative testing (invalid input) and regression testing of error handling.
- Test basis
- The expected behaviour: invalid input gets a clear error and no success flag.
- TechniqueError guessingWhere it showed upAn invalid methodology, a wrong value type and a malformed date sent to the snapshot endpoint, then the same kinds of input elsewhere.
- How we workedDefect clustering (a testing principle)Where it showed upThe same fail-open behaviour turned up in the mandates area first, then in performance snapshots, a third subsystem and a fourth. One fail-open endpoint usually means more.
Terms as in the ISTQB CTFL 4.0.1 syllabus and glossary.
What we found, and what it costs
- FindingPatternAn invalid methodology, value type or date answered 200 with the success flag set to trueWhat it costs youA monitor sees a healthy call and a client trusts the flag. Nobody is told that nothing happened.
- FindingPatternThe value field carried an internal exception class nameWhat it costs youDetail about the system's internals reaches callers who should only see a clear error.
- FindingPatternThe same behaviour in four subsystemsWhat it costs youEach endpoint validates on its own, so each is its own chance to fail open. Validation in one place that every endpoint passes through is the root fix.
The working behaviour was checked too: 12, 16, 9 and 14 checks passed in the four areas, so the client could see what was sound as well as what was not.
What changed
A list of fail-open endpoints, each with the request that proves it, and a witness test for every one that turns green on its own the day the fix lands. A suite that cannot run is reported as not reported.
Want your API tested for failures that look like success?
We test what an endpoint returns on bad input, not only on good input. The first month is a trial.
