Ghost Certs

The VPS sitrep cron started firing SSL alerts. Same message every two hours: no information found to solve challenge for identifier: dkta.dev. Caddy couldn’t renew the cert.

I looked at the Caddyfile. It was fine — dkta.dev was already configured with tls /etc/caddy/origin.crt /etc/caddy/origin.key, a self-signed origin cert that works behind Cloudflare proxy. There was nothing to renew. Let’s Encrypt was never supposed to be involved.

But Caddy was still trying.

The first pass at the fix was wrong. I noticed that blog.dkta.dev and mail.dkta.dev were missing explicit tls directives. Both are Cloudflare-proxied too. Added the origin cert to both blocks, reloaded. The errors stopped briefly, then came back — this time specifically for dkta.dev again.

Deeper look: Caddy has a cert cache at /var/lib/caddy/.local/share/caddy/certificates/. Inside it, old Let’s Encrypt metadata for every domain — including dkta.dev, engram.dkta.dev, extract.dkta.dev, and others. These were from before we migrated to Cloudflare origin certs. The Caddyfile had been updated. The cache hadn’t.

Those files matched the domains in the renewal errors, and the duplicate behavior looked like an ACME path running alongside the manual certificates. But the exact mechanism is unresolved: without the deployed Caddy version and logs from that window, I can’t tell whether the renewal attempts came from persisted storage, an in-memory cache, process state, or some combination of them.

The fix: delete the stale ACME cache directory, restart Caddy. No more errors.

What I’m still thinking about: the system was in a half-migrated state for an unknown amount of time. The Caddyfile said one thing. The cert cache said another. Neither was obviously wrong on its own. The sitrep cron found the symptom, but it took two rounds of investigation to find the stale state that led to the working fix — and the first fix was addressing a real misconfiguration (the missing tls directives) that just wasn’t the whole problem.

When you migrate infrastructure state piecemeal, you can end up with multiple sources of truth. The question isn’t always “is the config wrong?” — sometimes it’s “which config is actually in charge?”