Your API errors are a promise to your customers. On a signing API, none of the three held
For CTOs and API owners at fintech infrastructure companies whose customers build on their error responses.
If customers build screens on your error responses, a field that drifts breaks their product. A fintech infrastructure company had written down how its API errors must look. We called the running API and compared every error response, field by field, with that standard.
- Fintech infrastructure
- Transaction signing API
- Clients
- Reputation
- Money
What was at risk
Customers' front ends cannot render the errors
The standard promised a trace id, a structured list of field errors and a vocabulary of codes a front end can show. The running API met none of the three on any of 11 signing endpoints, so a customer's screen cannot show a trace id or a field error.
Messages that send users the wrong way
Key generation answered with an internal key-generation error when the real cause was authentication, and a rejected name returned one word, Invalid. A user who reads either is sent the wrong way.
A fix closed that had not held
One fix was accepted from a code read because a shared guard looked right. Our runtime pass found the vault-by-id route bypassing that guard. Closed on the read alone, the gap would have stayed open while the team believed it shut.
Standards and rules in play
- The company's own error standardA written design record promised three guarantees for every error. It was still marked provisional, so the report treats the result as conformance drift to confirm.
- Environment-variable validationA separate check ran 32 cases against the product's configuration rules and found no enforcement defects, so the report also says where the product was sound.
Context: financial firms in the EU answer to DORA for managing ICT incidents, and a trace id on every error is what makes an incident traceable.
How we tested it
- Test level
- System testing at API level, against the running service, followed by a retest.
- Test types
- Functional conformance testing against a written specification (black-box), then a retest of the 13 findings.
- Test basis
- The company's provisional design record for error responses, with 121 validation requirements across 11 signing endpoints.
- TechniqueEquivalence partitioningWhere it showed upFour error classes were probed, one from each partition: authentication, validation, not-found and malformed input. All four were non-conforming.
- How we workedConformance check against the written standardWhere it showed upEvery error response compared field by field with the three guarantees of the provisional design record.
- How we workedCode read, then runtime passWhere it showed upAt retest, 8 findings were confirmed fixed in code, 4 reopened and 1 was partial. Our runtime pass overturned one fix the code read had accepted, so the report recommends a runtime retest for the rest.
Terms as in the ISTQB CTFL 4.0.1 syllabus and glossary.
What we found, and what it costs
- FindingConformanceThe running API met none of the three envelope guarantees on any of 11 signing endpointsWhat it costs youFront ends cannot show customers a trace id, a field-level list or a stable code. One shared serialisation layer is behind it, so one change can repair every endpoint.
- FindingMessageKey generation reported an internal key-generation error when the real cause was authenticationWhat it costs youThe user is sent in the wrong direction and support gets the ticket.
- FindingMessageA rejected name returned the single word InvalidWhat it costs youA dead end: nothing says what to change.
- FindingReopenedA fix a code read had accepted did not hold when we ran it, because the vault-by-id route bypassed the shared guardWhat it costs youOur runtime pass caught what the read had accepted. Closed on the read, the gap would have stayed open and unreported.
A separate check of environment-variable validation ran 32 cases and found no enforcement defects.
What changed
At retest, 8 of the 13 findings were confirmed fixed in code, 4 reopened and 1 was partial. Our runtime pass overturned one fix the code read had accepted, so the report recommends a runtime retest to make the other verdicts definitive, and names the one shared layer behind the drift.
Want your API tested against its own documentation?
We run the endpoint and compare every field with your standard. The first month is a trial.
