When something crashes, you know within minutes. When something quietly does nothing and says it worked, you find out weeks later, from a customer or a missing result. After running several products ourselves, this is the bug shape that has cost us the most time.
Four ways it shows up
- The swallowed error. A scheduled deploy job fetched new code, and the fetch failed. The script ended with “exit successfully if the fetch fails”, so every run logged success while shipping nothing. It went unnoticed for about two weeks. The fix was to make the failure loud and to record which version is actually live.
- The status light that only reflects the past. A second-opinion reviewer stopped working after a tool upgrade, but its status badge stayed green because the badge only updated when a real job ran. A passive indicator says “the last thing worked”, not “it works now”.
- The partial artifact. An export feature that was supposed to hand a customer their whole project zipped only the built output folder, not the source. Everything looked fine until someone needed to edit it.
- The default that hides a block. When a page sat behind a login wall, the audit fell back to default values and reported a normal score. The score was real arithmetic on the wrong page.
The one rule
Probe the capability, never the artifact.
A log line, a green badge and an exit code are artifacts: evidence that something ran. The capability is whether the thing does its job right now. So a health check should perform a small real job end to end, such as asking the reviewer a trivial question and checking the answer, rather than reading a status field.
What we do now
- Guard clauses and fallback values get a second look. Every “or exit zero” and “or empty object” is a place a failure can hide.
- Deploys record the version they shipped, and a separate check compares it with the version that is live.
- Alerts fire on missing results, not only on errors. If the report did not appear, that is the alert.