On August 20 at 09:33 UTC our two scheduled production canaries stopped running. We did not find out for 25 hours, and when we did find out, the thing that had gone wrong was not production. Production was healthy the entire time. The canaries were the only casualty, and they failed in the one way that is genuinely difficult to notice, which is quietly.
The fix took an afternoon. The shape of the failure is the part worth writing up, because it is the same shape as the failure that puts identity teams in a bad position after an incident.
What actually happened
A run would appear in the list, marked as failed. The job underneath it had started and completed two seconds later with zero steps recorded. Pulling one of those jobs back up today still shows it: started at 19:32:57, completed at 19:32:59, steps recorded, none. There were no logs to read because no log had ever been written, and asking for the failed log returned a not-found. The provider status page said everything was operational, which it was.
The real message was in a place we had not thought to look, which is the check-run annotation rather than the run output:
The job was not started because recent account payments have failed or your spending limit needs to be increased.
That single sentence covers two quite different situations, a payment method that stopped working and an allowance that ran out, and it renders identically for both. The measured cadence makes the allowance the likely explanation in our case, since the canaries were consuming somewhere around 7,900 billed minutes a month against an allowance in the 2,000 to 3,000 range on a private repository, so the cost of watching the system had quietly outgrown the plan paying for it. The message itself does not tell you which one you are looking at, though, and only the billing settings page does.
While the canaries were red we ran the identical test files from a laptop against production, which came back nine passing and zero failing. That is the detail that reframed the whole incident. We were not looking at an outage. We were looking at the absence of a signal, and we had spent the first stretch of the investigation assuming those were the same thing.
It was the second time in two weeks
On August 6 the same two canaries went blind for an unrelated reason, which was a multi-hour outage at the CI provider itself. Both jobs failed at the very first step, before a single line of our test code ran, because the runner could not download its own checkout action from the provider's CDN. Our edge observability confirmed the consequence directly, which was that the signup canary made zero requests against the login endpoint for roughly four hours. Production was fine that time too.
Two unrelated causes, two weeks apart, one identical symptom. In both cases the safety net was gone, and the dashboard did not look alarming, it looked calm.
The property that makes this hard
A monitor that has stopped running and a system that has nothing to report produce the same artifact. Alerting is built almost entirely around things arriving, so the absence of arrivals reads as good news right up until the moment you check.
This is not really a CI problem. It is a property of any check that lives outside the thing it checks and depends on a third party's continued willingness to run it. Scheduled jobs, uptime monitors, notification pipelines and nightly backups all share it. Each can be switched off by an event that has nothing to do with your software, including an expired credential, a revoked permission, a changed quota or an unpaid invoice, and none of those events announce themselves in the place you are actually watching.
What we changed
We moved the probes into the product. Every check the canaries ran against production was already a plain HTTPS request with no filesystem access, no child process and no runtime-specific dependencies, so the logic ported to our Cloudflare Worker unchanged and now rides the fifteen-minute cron trigger the project already runs. Same cadence, no metered CI minutes, and no dependency on a third party agreeing to start a job.
The decision logic sits in pure, dependency-injected functions with no side-effectful imports, which is the same split we use for our rate limiter and cron routing. The deployed Worker and the Node test runner execute the identical code path, so a test passing while production behaves differently is structurally impossible rather than merely unlikely. The probes cover the failures we have actually been bitten by, which are secret parity, the authenticated route contract, the signup pipeline end to end, the send-failure log and the mail provider key.
One thing did surprise us on the first live run. A Worker cannot fetch its own public hostname, because the subrequest loops back out through the edge and times out, which we saw as an HTTP 522 and briefly misread as a real outage. Same-origin probes now route through a service binding that points the Worker at itself internally.
In the 24 hours before publishing this, the canary completed 96 runs out of 96 scheduled, which is the full four-per-hour cadence with nothing dropped, and the most recent run finished about a minute before I checked.
The blind spot we did not close
Moving a check inside the thing it checks buys reliability and costs coverage, and it would be dishonest to write this up without naming the coverage it costs. These probes cannot detect that the Worker is entirely down, for the obvious reason that a dead Worker runs no cron. That case needs an external pinger against our public health endpoint, and it is written into the source as a known and deliberate gap rather than left in place for someone to trip over later.
What self-hosting does cover is the far more common class, which is a missing secret, a revoked key, a schema drift or a broken auth contract. Those are the outages a service can accurately observe about itself, and reading bindings from inside the runtime is strictly more accurate than asking a control plane what it believes it deployed.
Why this matters if you run identity
The reason I am writing this on a company blog rather than in a private postmortem is that most identity teams are running the same bet somewhere, usually on backups.
A nightly export job that reports success is making a claim about the job. It is not making a claim about whether you can rebuild your policies, your app assignments, your group rules and your lifecycle automations into a working org while an incident is live. Those are separate questions, and the gap between them tends to stay invisible for exactly as long as nobody tests it, which in practice means until the worst possible afternoon.
The questions worth asking about your own setup are fairly plain. If the backup job stopped running entirely, how would you find out, and how long would that take. Does anything in your alerting distinguish a quiet system from a stopped one. When did someone last restore from a backup into a state they could actually inspect, rather than reading that the job went green. If the honest answer to the last one is that nobody has, that is the same silence we spent 25 hours reading as health.
Butterfly Security exists for that gap. We keep versioned backups of your identity configuration, we show you a restore preview and a diff before anything is written, and we score restore readiness per connection so the number moves when your recoverability actually changes rather than when a job merely completes. Okta, Okta Workflows, Auth0, Microsoft Entra ID and Google Workspace are supported today for connect, backup, restore and diff.
If you want to see it against your own org, start a 30-day free trial or book 30 minutes with me and I will walk you through a restore preview on your configuration.