The scanner that always passed
I built a security gate for Docker images, then found it reporting PASSED while scanning nothing at all. This is how I proved it, traced it, and made it fail closed.
What it was supposed to do
The tool checks a Docker image before it reaches production. It looks for three classes of problem: known vulnerabilities in the packages the image ships, credentials and API keys accidentally baked into the code, and unsafe settings in the Dockerfile itself. If it finds something serious, it blocks the build.
That last sentence is the entire value of the thing. A scanner that reports problems is a report. A scanner that stops the build is a gate. Everything else — the formatting, the speed, the number of tools wired together — is detail around that one job.
The bug
The gate had a failure mode where it reported PASSED without having scanned anything.
When the underlying scanning tools failed to start, the pipeline did not error. It did not warn. It returned a green light and let the build through, exactly as though the image had been examined and found clean.
I proved it by running the tool against a test image I had deliberately filled with problems: 22 critical vulnerabilities and 5 exposed AWS and GitHub keys. It passed. Every one of those findings was sitting in the image, and the gate waved it through.
A gate that fails open is worse than no gate, because everyone downstream believes it is working.
The root cause
The check counted the problems it found, and never asked whether the scan had actually run.
Those two things look identical from the inside. Finding zero problems and never looking for any both produce an empty list of findings, and the gate treated an empty list as success. The missing question was not "how many problems are there" but "did I successfully look".
Three different conditions could trigger it. Two of them came from the project's own setup instructions — anyone following the README could land in the broken state without knowing. The third made the tool stop working on Windows entirely, silently, with the same green result.
The fix
The gate now fails closed. If any scan cannot run, the build is blocked, and the report names the exact tool that failed rather than reporting a generic error. The absence of findings is no longer evidence of safety.
Two related problems surfaced while fixing it. The result was being reported three different ways at once, which meant the three paths could disagree; those are now a single source of truth. And the user's own severity settings had no effect on the outcome — a single low-risk issue failed every build regardless of configuration.
That second one matters more than it looks. A check that is permanently red is a check that gets switched off, and a gate nobody runs protects nothing. Making the thresholds real is what turned it from an obstacle into something a team would actually keep enabled.
| Before | After |
|---|---|
| Reported PASSED | Build blocked |
| No indication anything was wrong | Report names the tool that failed |
| Three conflicting result paths | One source of truth |
| Any low-risk finding failed the build | User severity thresholds decide |
Finishing it properly
Finding the bug was most of the work, but a tool nobody can install is not shipped. The automated pipeline had never once passed — 85 attempts, 85 failures — and now passes all five checks.
Test coverage went from 63% to 96.5% across 272 tests, so each fix is regression-tested on its own rather than being verified by hand once and hoped about afterwards. Install dependencies dropped from 24 to 5, the generated report from 8.7 MB to 1.6 MB, and the Docker build from 237 MB to 380 KB.
What I took from it
Test the failure path, not just the success path. I had tested that the scanner caught problems. I had never tested what it did when the scanner itself broke, and that was the only case that mattered.
Absence of evidence is not evidence of absence — a principle that sounds obvious written down and is very easy to encode into a boolean by accident. Any check that reports success should be able to say why it believes that, not merely that it found nothing.
And read your own setup instructions as an adversary would. Two of the three ways to trigger this came from the README I wrote myself.