CI Found It. Git Had Already Kept It.
The important part of a secret-scanning failure is not the red CI badge.
on this page
The important part of a secret-scanning failure is not the red CI badge.
It is the part that happened before the badge showed up.
You push a commit. The remote accepts it. A secret scanner runs a minute or two later and flags a token, key, or credential string. The merge is blocked, the branch is noisy, and everyone feels like the system did its job.
It did part of the job. It did not prevent the secret from entering history.
That distinction matters because the cleanup burden starts the moment the push succeeds, not the moment CI reports a failure. Once the remote has the commit object, the question changes from “did we catch it?” to “who can still reach it, for how long, and what do we have to rotate now?”
The Timeline
The confusing bit is that the workflow looks safe from the outside.
You commit locally. You push to the remote. CI starts. CI scans the diff. CI blocks the merge.
That sequence feels like a prevention story because the secret never lands on main.
But the remote already wrote the commit object before the scanner ever ran. The branch may be blocked, the PR may be closed, and the code may never merge, but the secret still existed in a repository object the remote accepted.
That is why this class of failure is easy to underestimate. The visible outcome is “merge denied.” The operational outcome is “rotate the secret and check where else the old value leaked.”
Those are not the same thing.
Where The Defenses Sit
Not every control is trying to stop the same step.
Pre-commit hooks run on your machine. They stop the bad commit before git commit finishes.
Pre-receive hooks and push protection sit on the remote. They can reject the push before the commit object is accepted.
CI runs after the push. It can alert, block merges, and force remediation, but it is already downstream of the write.
That is the real hierarchy:
- local hooks stop the write before it leaves your machine
- remote hooks stop the write before the server accepts it
- CI detects the problem after the server already has the object
If you have server-side push protection, great. That is the right place to intercept secrets.
If you do not, CI is still worth having, because a late detection is better than no detection. It just needs to be understood honestly. It is not a valve. It is a pressure gauge.
What Actually Happens After The Push
The push itself is fast:
- Git packages your local commits into objects.
- The remote writes those objects to its storage.
- The remote replies success.
- CI wakes up later and scans what was already written.
That order is the whole problem.
Once the commit exists remotely, any delay before CI fires becomes exposure time. On a quiet runner queue that might be a short window. On a busy system it might be long enough for another developer, a bot, or a fork to see the object before the scanner finishes.
That is why “the PR was blocked” is not the same as “the secret never existed outside my laptop.”
It existed. The scanner just arrived after the fact.
For a public repository, that is an obvious problem. For a private repository, the risk is narrower but not zero. Anyone with read access and the commit SHA may still be able to retrieve the object. Mirrors, forks, backups, and automated tooling can also preserve the leaked value in places you do not immediately think to check.
The question is not whether the exposure is identical across every repo type. The question is whether you want to rely on a late detector to clean up a problem that could have been stopped earlier.
Why Rotation Is Annoying
Rotation is the tax you pay for letting the secret reach history.
If the token was used by backups, deployment scripts, container jobs, CI variables, or a homelab service account, the cleanup is no longer a single change. It is a dependency trace.
That is the difference between catching a bad secret in pre-commit and catching it after push:
- pre-commit rejection costs you almost nothing
- CI detection costs you a rotation, a verification pass, and some manual archaeology
The annoying part is not the act of changing the token. It is proving that every consumer of the old value was found and updated in the right order.
If you rotate too early, something fails unexpectedly. If you rotate too late, the old secret keeps working somewhere you forgot. If you miss one environment variable in one repo, the cleanup is fake.
That is why the right fix is to move the catch as far left as possible.
The Layered Defense
The best version of this stack is boring:
pre-commitstops obvious secrets before the commit is created- optional server-side protection rejects secrets before the remote stores them
- CI still scans as a backstop and alerts on anything that escaped the first two layers
That is not redundancy for its own sake. It is defense in depth.
Each layer protects a different boundary. The local hook protects your workstation. The remote guard protects the repository. CI protects the review and delivery pipeline.
If you only have one of them, be honest about which one it is.
In a lot of real systems, especially homelabs and smaller teams, CI is the only layer people actually have. That makes it valuable, but it also means the team needs to treat secret scanning as detection and rotation, not prevention and absolution.
The Small Config That Saves You The Big Cleanup
The most practical move is still the simplest one.
repos:
- repo: https://github.com/Yelp/detect-secrets
rev: v1.5.0
hooks:
- id: detect-secrets
args: ["--baseline", ".secrets.baseline"]Run pre-commit install in each clone. Let the hook block the commit before it leaves the laptop. Keep the baseline file audited so you are not training people to ignore false positives.
That one file is worth more than a thousand words of “just be careful.”
It does not make the system perfect. It does make the common failure mode cheap.
The Real Lesson
The point is not that CI is useless.
CI is useful. It catches real mistakes. It prevents bad code from merging. It gives you a place to enforce policy after the fact.
The point is that a late detector should not be mistaken for a preventive control.
If the secret reached the remote, you already crossed the expensive line. The scanner may still save you from further spread, but the cleanup is now real work.
So the right question is not “did CI catch it?”
The better question is:
Did we stop it before the remote had a chance to keep it?
If the answer is no, then the next job is rotation, not celebration.
CI blocked the merge. The secret had been in git for ninety seconds.
CI blocked the merge. The secret had been in git for ninety seconds.