The Certificate Authority Compromise That Reached Docker AI Sandboxes

Docker Sandboxes shipped a guidance page that sysadmins should read closely, because it is quietly a lesson in how TLS trust actually works: after the eMudhra intermediate CA compromise, Docker told users not just to update the sandbox CLI, but to recreate sandboxes from refreshed images and to update CA certificates inside sandboxes that already exist. If your first instinct is "but I already ran the updater", read on. The updater updated the wrong thing.

This is a natural companion to our look at sandboxing AI coding agents in Docker and our agent egress firewall piece: both deal with keeping untrusted agent code contained. This one deals with the opposite direction: who the sandbox itself is willing to believe, and what happens when someone it believes turns out to be untrustworthy.

What happened with eMudhra

eMudhra is a commercial certificate authority whose emSign roots were included in the major public trust stores. At a key generation ceremony in April 2025, a script error in a key wrap/unwrap step produced a second, weak set of keys alongside the properly generated ones, and the weak keys are what ended up in eight intermediate CA certificates. Nobody stole anything, and there is no evidence eMudhra was breached: the keys were so weak that anyone could recover them from the public certificates. Every one of the eight is a number below 256, and one of them is literally 4. A properly generated key should take roughly 10^115 guesses to find. Six of the eight were TLS intermediates trusted by the Apple, Chrome, Microsoft, and Mozilla root programs; the other two were Microsoft-trusted code-signing CAs. After disclosure on October 17, 2026, the fallout included removing eMudhra's emSign ECC Root CA - G3 from Mozilla's and Chrome's trust stores, and eMudhra announcing it is exiting the global SSL/TLS certificate business.

Cloudflare's incident writeup, the clearest public account, splits the two things this article keeps separating: their own response was updating shared trust stores, base images, and other shared foundations, paired with the reloads, restarts, and image deployments needed to activate them. Their guidance to developers is to apply the relevant dependency updates, rebuild affected images, and work out whether each application needs a reload, restart, or full redeployment to actually pick the new trust state up.

Why does a weak intermediate key matter? Because of how chain of trust works, and it is worth spelling out for people who mostly see TLS as "the padlock in the browser".

The TLS chain of trust, in operator terms

TLS identity verification is not "the site's certificate is valid". It is a chain: your system holds a small set of root CA certificates (the trust store), the server presents its certificate plus its intermediates, and your TLS library walks the chain: server certificate signed by an intermediate, intermediate signed by a root that is in your trust store, with key usage and basic constraints permitting each signature, validity dates in range, and revocation not disqualifying it. The root itself is self-signed; it does not need to prove anything upward, because being in your trust store is its authorization.

Now the crucial asymmetry: an intermediate CA is not the root. A compromised leaf key endangers one hostname at a time. A compromised intermediate key is operationally much worse: whoever holds one of the six TLS intermediates can mint certificates for any hostname, and those certificates chain through the compromised intermediate to a root your systems already trust (the two code-signing CAs are arguably worse: they could sign malware that validates under any publisher name). Your machine is not misconfigured. Its configuration faithfully says "I trust eMudhra's root", and the attacker's certificate honestly does chain to that root. The fix therefore cannot come from your side re-verifying anything: the trust community has to make the chain stop verifying. Browsers did that within hours by blocklisting the intermediates through revocation mechanisms like OneCRL and CRLSets. But most other software that relies on web PKI has no mechanism to filter intermediate CAs, so the fix that actually reached server-side trust bundles was blunter: remove the emSign ECC Root CA - G3 root itself, which is what Mozilla did in NSS and Google did in the Chrome Root Store. Either way, your machines have to receive that decision before the math protects you again.

Detection mattered here, and it worked: the weak keys were discovered in October 2026 and the intermediates were blocked across the public root programs quickly (the certificates dated from April 2025, so the window was long). But note what detection does not do: it does not reach frozen copies of the trust store inside artifacts you already built.

Image trust stores: the bundle is baked, not linked

Your browser gets its trust store from a shared OS-level store (or ships its own), and it updates itself constantly. But a container image is not a browser. When you docker build, the base image snapshot contains a ca-certificates package as it existed on build day, and that package includes the CA bundle compiled at whatever date the distro shipped it. That bundle is a file, layered into the image, frozen until someone rebuilds or updates it.

The consequence is exactly what Docker and the security vendors flagged:

• An image built before the fix still trusts the compromised chain, no matter how patched the host is.
• Every image built FROM an old base image inherits the same stale bundle, so "but we rebuild from base every week" only helps if the base itself has been rebuilt with updated ca-certificates.
• A running container never updates at all. The point Aikido and Endor Labs both hammer: a running container keeps the old bundle until someone rebuilds or explicitly updates it. As Aikido puts it, a Docker image freezes its CA bundle at build time, so every image built before the fix, and every image built from an old base layer since, still trusts the compromised intermediates until someone rebuilds it.

The check that pulls you out of "we're probably fine" is boring and worth scripting:

# Debian / Ubuntu
dpkg -s ca-certificates | grep Version
# Alpine
apk info ca-certificates
# Fedora / RHEL
rpm -q ca-certificates

If the installed version predates your distro's fixed package, the image is stale. Do not eyeball a single version cutoff, though: distributions patched at different versions (Ubuntu ships 20260816ubuntu0.*, Debian 12 20250419~deb12u2, Alpine 20261020-r0, Fedora 2026.2.92_v9.0.319), so one threshold misclassifies some distros. The community incident site keeps a per-distribution and per-runtime table, so check yours against it. docker build --pull --no-cache forces the base image and package layers to refresh so the updated bundle is baked in, and your rebuild-deploy cycle then carries it to everything you run.

Stale CA bundles: the failure mode that purrs quietly

Here is what makes this class of problem sysadmin-dangerous: it does not look broken. An image with a bundle that is months too old passes TLS checks all day, connects to every registry and API endpoint, and behaves normally. There is no alarm, no error, no degraded function. The only difference between that image and a patched one is invisible: which certificates it would accept as valid.

That is why a stale CA bundle is not a compliance nicety after a CA compromise. The bundle is effectively a security policy file listing who can impersonate whom. An outdated bundle is a policy that still authorizes the attacker's chain. If your sandbox images fetch packages, pull models, call APIs, or clone repos (and AI agent sandboxes do all of those), the stale bundle means a network attacker with the compromised intermediates could present certificates your sandbox willingly accepts.

One honest scoping note, though: the eight intermediates never appeared in certificate transparency logs before they were blocked, so clients that enforce CT (Firefox, Chrome, Edge, and Safari) could not actually have been attacked with them. The realistic exposure is exactly the non-browser clients this article keeps pointing at: curl, git, language runtimes, CI, and container images, most of which do not enforce CT. That is the quiet part.

The other way bundles go stale is subtler: apps that bundle their own stores. OpenSSL using the distro store gets patched by the package above, but many runtimes and tools carry their own stash: certifi for Python, Node's own bundled store, the JDK's cacerts keystore, and custom REQUESTS_CA_BUNDLE / SSL_CERT_FILE paths in CI configs. Patching ca-certificates touches none of them. The Endor Labs and Cycode writeups both call this out as where most residual exposure lives: container images, language runtimes, and custom CA bundles keep trusting the compromised intermediates until someone updates or rebuilds each one explicitly.

Why updating the CLI did not update the environments it manages

Docker's guidance makes the distinction crisply, and it generalizes to every container-orchestration CLI you use.

Updating the sbx CLI (or Docker Desktop, or any manager) does three things: it gets the manager its own binary updated (so its bundled dependencies refresh), it can add new flags and behaviors (Docker's guidance, for example, is to pass --pull always yourself when creating sandboxes from built-in kits: a flag to use, not a new default), and it makes future operations use updated images. It does not, and structurally cannot, retroactively mutate already-created runtime environments. Docker's own announcement states it plainly: pulling updated images does not change sandboxes that already exist, so the guidance is to recreate sandboxes from the updated image, and meanwhile update the CA certificates inside existing ones.

Their concrete sequence from the announcement. Replace codex with your agent and restore any options you originally created the sandbox with, such as --clone or additional kits. And read the warnings first: removing a sandbox deletes its filesystem, so save any files you need before sbx rm, including unpushed commits in clone mode.

# Preferred: recreate each sandbox from the updated image
$ sbx stop <sandbox-name>
$ sbx rm <sandbox-name>
$ sbx create --name <sandbox-name> --pull always codex <workspace-path>

# If you must keep an existing sandbox alive: patch it in place
$ sbx exec <sandbox-name> -- sudo apt-get update
$ sbx exec <sandbox-name> -- sudo apt-get install --only-upgrade -y ca-certificates

Restart the applications running in the sandbox after updating: long-lived processes hold trust data loaded at startup, and a patched bundle on disk does nothing for a process that never re-reads it.

That distinction, the manager and the managed environment are different trust domains with different update lifecycles, is the actual lesson. A CLI is a thin client over state that lives elsewhere: images in a registry (immutable by design) and containers whose filesystem is a snapshot of an image at creation time. Updating the thin client is necessary and wildly insufficient.

This generalizes beyond Docker's sandboxes:

• Kubernetes: an upgraded kubectl does nothing to node images or running pods. Node images, and any custom CA bundle mounted into pods, are separate artifacts with their own refresh paths.
• Toolbx and devcontainers: those containers carry their own OS userland and bundle; they do not inherit your host's freshness.
• Digest-pinned builds: FROM image@sha256:... pinning (which I otherwise recommend for supply-chain integrity) actively preserves a stale bundle until you move the digest. The fix is moving the base image, since reproducibility itself is not the bug.
• Any custom bundle flag: everywhere you set NODE_EXTRA_CA_CERTS, REQUESTS_CA_BUNDLE, or ssl_cert_file, you have created your own frozen trust store with its own update problem.

What a sysadmin actually does, in order

1. Read the primary sources first: Docker's security announcements page for sandbox-specific steps, Cloudflare's writeup for the incident background, the community incident site for per-distro patched versions and test endpoints, and the Endor Labs, Cycode, and Aikido writeups for scan-and-rebuild checklists.
2. Inventory the trust surfaces: the OS ca-certificates package in base images, plus language-runtime stores (certifi, JDK cacerts, Node), plus any custom bundle flags in Dockerfiles and manifests.
3. Rebuild from fresh bases: docker build --pull --no-cache so updated bundles are baked in, not layered on guessed ancestry.
4. Redeploy, do not just rebuild: an image sitting in a registry protects nobody. Running containers, scheduled jobs, and CI runners holding cached old images are the real surfaces. Where a recreate is not possible this hour, patch in place (as in Docker's sbx exec sequence) and schedule the recreate.
5. Check the apps that do not auto-reload: some long-lived processes cache trust data at startup. That is why Cloudflare paired updates to shared trust stores, base images, and other shared foundations with reloads, restarts, and image deployments. Know which of your services need a restart to pick up new trust data.
6. Make CA-bundle freshness a standing property: a scheduled rebuild-with-pull, plus a build-time check on the ca-certificates version, turns this from a one-time scramble into a standing control. As the Cycode writeup puts it, moving the base image and removing or updating custom bundles is part of the fix, not an afterthought.

FAQ

Does a compromised intermediate CA let attackers decrypt my traffic?

No. The compromise enables impersonation: minting valid-looking certificates for arbitrary hostnames that chain to a root your systems trust. The practical risk is an active man-in-the-middle against a machine that would accept the forged chain. Updating the trust store closes it going forward, and it is not a reason to rotate your own application keys. The exception runs the other way: if you operate servers whose certificates chain to emSign ECC Root CA - G3, Google's guidance is to immediately acquire and install a new TLS certificate from another PKI hierarchy. The root removal breaks your existing chain, and with eMudhra exiting the TLS business you should plan the migration rather than wait for renewal.

If I update ca-certificates in a running container, am I safe?

Mostly, for that container, for the OS-level TLS library. Two caveats. Runtimes or apps with their own bundle (certifi, JDK cacerts, custom CA_BUNDLE-style paths) each need their own update. And some long-lived processes hold trust configuration loaded at startup, so a restart is often needed regardless. The durable fix is rebuilding the image so future containers start patched.

Why not just delete the compromised intermediates from images by hand?

There is nothing to delete. A typical PEM bundle (and the ca-certificates package behind it) contains roots, not these intermediates. The eight were never in your bundle, and an attacker presents the intermediate themselves during the handshake. That is exactly why most non-browser software could not be fixed by blocklisting intermediates the way browsers did, and why the shipped fixes remove the emSign ECC Root CA - G3 root instead. If you need an interim measure before the patched package lands, remove that root, or use your platform's distrust mechanism if it has one. Treat any hand-edit as a stopgap, because package updates and the next build will overwrite it.

How long does a rebuilt image stay safe?

Until the next root-program change it should reflect, which is the whole point: CA bundle freshness is a standing property, not a one-time check. Root programs add, revoke, and distrust CAs continuously. The operational fix is a scheduled rebuild-and-redeploy cycle plus build-time checks on bundle version, so the next incident does not depend on someone watching advisories.

Further reading

• Docker security announcements, including the Docker Sandboxes response to the eMudhra CA compromise: https://docs.docker.com/security/security-announcements/
• Cloudflare, "8 eMudhra certificates are compromised": https://blog.cloudflare.com/emudhra-certificate-compromise
• Google, "Protecting Chrome and Android users from compromised eMudhra intermediate CA certificates": https://blog.google/security/protecting-chrome-and-android-users-from-compromised-emudhra-intermediate-ca-certificates/
• Community incident site, per-distro patched versions and test endpoints: https://emudhra.we-have-a-problem.com/
• eMudhra statement on the incident: https://emudhra.com/en/post/certificate-security-incident
• Endor Labs, "eMudhra intermediate CA compromise, the trust stores you patched and the ones you didn't know you had": https://www.endorlabs.com/learn/emudhra-intermediate-ca-compromise-the-trust-stores-you-patched-and-the-ones-you-didnt-know-you-had
• Aikido Security, "eMudhra CA compromise, eight weak private keys": https://www.aikido.dev/blog/emudhra-ca-compromise
• Cycode, "eMudhra CA compromise, what to patch and how to find it": https://cycode.com/blog/emudhra-intermediate-ca-compromise/
• Our write-up on sandboxing AI coding agents in Docker: https://systhoughts.com/posts/docker-sandboxes-ai-agents
• Our write-up on egress firewalls for AI coding agents: https://systhoughts.com/posts/ai-coding-agents-egress-firewall

In your estate, who actually owns the freshness of the trust stores baked into your sandbox and CI images: the platform team, each app team, or (the honest default) nobody? If this incident made you go check, say so in the comments.

Until next time, keep your systems thoughtful.