mTLS for Internal Services: Useful Security or Homelab Theatre, Hands-On

If you have been following the security conversation on this blog, you know the pattern: the Fail2ban vs CrowdSec post settled who blocks at the network edge, the reverse proxy comparison showed where Caddy, Traefik, and Nginx Proxy Manager terminate TLS, and the Docker secrets post made the case that every layer only earns its keep when it matches a real threat model.

mTLS is the next layer in that conversation, and it is the one where the marketing and the reality drift apart the fastest. This post is the sysadmin version: what mTLS actually proves, the threat model it fits, hands-on client certificate configuration for Caddy and Nginx, what a service mesh changes, and the mobile and unsupported apps that quietly decide whether the whole exercise is security or theatre.

The short version

• mTLS proves who is talking, not just who you are talking to. Regular TLS authenticates the server to the client. mTLS adds a client certificate, so the server also authenticates the caller. That one change turns an internal API from open to identity-gated.
• It fits one threat model well: anything that can reach the port can no longer impersonate a service. A device on the LAN, a guest network, a compromised IoT box, or an attacker already inside the VPN has to hold a valid certificate before your services answer.
• It does not fix application bugs, it does not stop a compromised service that holds its own key, and it does not survive root on the host. mTLS is transport authentication, not application security. The certificate and private key are only as safe as the box that holds them.
• The hard part is key distribution and renewal, not the handshake. A client CA, a certificate per service, and a rotation story are the actual work. Skip that and the certificate expiry is your outage, not your protection.
• Mobile and unsupported apps are the wall. Browsers can present client certificates, most native apps cannot, and IoT devices have no user interface for it at all. Dual mode and overlay VPNs are the honest answers there.
• Caddy and Nginx cover the self-hosted case; a service mesh is for Kubernetes. Client authentication at the reverse proxy is a weekend project. Linkerd and Istio add per-workload identity when the cluster is already real, which the Compose vs Kubernetes post says is the point where a cluster earns its keep.

What mTLS actually is

TLS authenticates one side. When your browser connects to a site over HTTPS, the server proves its identity with a certificate, and the client mostly stays anonymous. That is one-way authentication, and it is correct for the public web.

mTLS, short for mutual TLS, simply authenticates the other side too: the client presents its own certificate, and the server verifies it during the handshake. In TLS 1.3, RFC 8446, the client certificate is encrypted inside the handshake, so it is not visible to someone sniffing the wire.

The practical shape is a small PKI. You run a client CA, issue a certificate to each service or device that is allowed to call in, and configure the server to trust that CA rather than individual certificates. When a client connects, the server checks the chain, the expiry, and optionally a revocation list, and only then lets the handshake complete.

The certificate becomes the identity: the common name or SAN is who the caller claims to be, and the private key is the proof. If every service shares one certificate, mTLS is mostly a shared secret with extra steps; the value shows up when identities are per service or per device.

The threat model that matters

Start from where self-hosted services actually sit. Behind a reverse proxy that terminates TLS, or worse, bound to every interface on a home LAN. The people who reach those services are not necessarily hostile, but the network is not empty either: guests, IoT devices, a compromised laptop, a misrouted VPN. The realistic question is not whether the internet can reach you, it is whether a host that is already on the network can reach everything. That is the threat model mTLS answers.

What it protects. The list is short and precise:

• Sniffing. Internal traffic gets real TLS encryption, so a passive listener on the LAN or the VPN cannot read the API calls.
• Spoofing and impersonation. A host that can reach the port but holds no valid client certificate gets a failed handshake, not a request. Port scanning stops returning useful answers, and a device on the guest network cannot call the admin API.
• Lateral movement from an unauthenticated foothold. If an attacker controls one laptop on the network, they still need a certificate before any mTLS-protected service answers. That is a real barrier, not a speed bump.
• Per-caller identity. The proxy can log which certificate called which endpoint and can gate routes per certificate or per subject. That is audit data you do not get from an IP allowlist.
• A clean story for server-to-server calls. Cron jobs, backups, and internal APIs can authenticate as themselves instead of embedding a shared token in a script.

What it definitely does not protect. Say this list out loud before you add the PKI:

• Application-layer attacks. SQL injection, XSS, auth bypass, RCE inside the app. mTLS does not look at the request, it only gates the connection. The Naxsi and CrowdSec layers still belong in the stack.
• A compromised service that legitimately holds a key. The moment one service is owned, the attacker inherits its certificate and calls everything that service was allowed to call. This is the biggest gap, and it is exactly why meshes rotate short-lived identities and why you need a rotation story too.
• Root on the host. Root reads the private key from disk and impersonates the service. The Docker secrets post made this point for secrets, and it applies verbatim to certificate keys.
• Insiders and valid-cert holders. Anyone with a legitimately issued certificate is indistinguishable from a legitimate caller. mTLS authenticates, it does not authorize intent.
• SSRF. A service with a certificate can call other services, and that traffic looks perfectly normal to mTLS. The filter has to happen at the application layer.
• DoS and abuse. There is no rate limiting and no WAF in a handshake. A client with a valid cert can hammer an endpoint all day.

At a glance: what mTLS protects and what it does not

Hands-on: Caddy

Caddy has a client_auth directive inside the tls block, and it is the cleanest way to add mTLS to a self-hosted stack. The modes are request, require, verify_if_given, and require_and_verify. Require a certificate and verify it, with a trust pool that points at your client CA:

internal.example.com {
	tls {
		client_auth {
			mode require_and_verify
			trust_pool file {
				pem_file /etc/caddy/client-ca.pem
			}
		}
	}
	reverse_proxy 127.0.0.1:8080
}

Caddy exposes the verified client in placeholders, which is how you log or gate on identity. The full forms are {http.request.tls.client.subject}, {http.request.tls.client.issuer}, and {http.request.tls.client.fingerprint}; recent versions also accept the shorter {tls_client_subject}. Forwarding them as headers gives the app behind the proxy the same identity data the handshake proved:

internal.example.com {
	tls {
		client_auth {
			mode require_and_verify
			trust_pool file {
				pem_file /etc/caddy/client-ca.pem
			}
		}
	}
	reverse_proxy 127.0.0.1:8080 {
		header_up X-Client-Subject {http.request.tls.client.subject}
		header_up X-Client-Fingerprint {http.request.tls.client.fingerprint}
	}
}

$ sudo caddy validate --config /etc/caddy/Caddyfile
Valid configuration

$ sudo systemctl reload caddy

# a client without a certificate now gets a TLS alert instead of a page
$ curl -sv https://internal.example.com/ 2>&1 | tail -n 5

Hands-on: Nginx

Nginx has supported client certificate authentication for decades through the ssl module. Three directives do the work: ssl_client_certificate points at the trusted CA bundle, ssl_verify_client decides the policy, and ssl_verify_depth limits the chain length. A strict server block:

server {
    listen 443 ssl;
    server_name internal.example.com;

    ssl_certificate     /etc/ssl/internal.example.com.pem;
    ssl_certificate_key /etc/ssl/internal.example.com.key;

    ssl_client_certificate /etc/nginx/client-ca.pem;
    ssl_verify_client      on;
    ssl_verify_depth       2;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header X-Client-Verify $ssl_client_verify;
        proxy_set_header X-Client-DN      $ssl_client_s_dn;
    }
}

ssl_verify_client accepts on, off, optional, and optional_no_ca (requests the client certificate but does not require it to be signed by a trusted CA certificate). on rejects clients without a valid certificate at the handshake. optional requests a certificate and verifies it when one is provided, which is the dual-mode pattern for mixed clients: the $ssl_client_verify variable is SUCCESS for verified callers and NONE for clients that did not present one. A map turns that variable into a readable access decision:

map $ssl_client_verify $client_ok {
    default      0;
    SUCCESS      1;
}

server {
    listen 443 ssl;
    ssl_client_certificate /etc/nginx/client-ca.pem;
    ssl_verify_client      optional;
    # ...
    location /internal {
        if ($client_ok != 1) {
            return 403;
        }
        proxy_pass http://127.0.0.1:8080;
    }
}

Service mesh: identity at the pod layer

If you run a real Kubernetes cluster, the Compose vs Kubernetes post drew the line: the cluster earns its keep when a second host exists (probably more, actually) and the platform becomes the point. That is also where the service mesh earns its keep.

Linkerd turns on mTLS by default between meshed workloads and derives identity from Kubernetes service accounts, so each pod gets its own short-lived certificate that rotates automatically. Istio does the same with PeerAuthentication, where the STRICT mode refuses plaintext:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: apps
spec:
  mtls:
    mode: STRICT

The mesh solves the two problems that make hand-rolled mTLS theatre: key distribution and rotation. Identity is injected with the workload, certificates are short-lived and rotated by the control plane, and per-workload identity means a compromised pod does not inherit a long-lived shared credential. The cost is the mesh itself: control plane components, sidecars or ambient mode, and an upgrade cadence. On a Compose stack, that is exactly the second job the Compose vs Kubernetes post warned about. The honest ordering is: proxy mTLS on the host for a few services, mesh when the cluster is real and the workloads are many.

Mobile and unsupported apps: the wall

Here is where strict mTLS stops being a security decision and becomes a compatibility decision. Browsers can present client certificates: iOS Safari and Android Chrome will use a certificate from the device trust store, with a picker and occasional re-prompts. The experience is rough, especially with hardware-backed keys, but it works. The problem is everything else.

Most native apps do not expose client certificate configuration at all. A Jellyfin or Plex client, a photo backup app, a Nextcloud sync client, a chat app: the HTTP stack either supports client certificates internally or it does not, and the app rarely gives you the option. IoT devices, printers, and streaming boxes have no user interface for it. Enable strict mTLS in front of those and they simply break, which is exactly the homelab theatre this post is about.

• Dual mode. Use ssl_verify_client optional on Nginx or verify_if_given on Caddy. Server-to-server callers present a certificate and get the strict path; apps and humans without one fall through to app-level authentication. One listener, two tiers.
• App-level auth for humans. Keep OAuth2, API keys, or a login in front of user-facing clients, and reserve mTLS for the service-to-service paths that can hold a certificate. This is the pattern the Docker secrets post applied to credentials: match the mechanism to the client.
• Overlay VPN for device fleets. Tailscale, NetBird, or a plain WireGuard mesh gives every node authenticated, encrypted transport without touching each app, which the NetBird vs Pangolin post covers in depth. The trade: you get transport-level trust between machines, not per-service identity at the application layer. For a house full of mobile devices, that is usually the right trade.

The decision rule

• Use proxy mTLS when: services sit behind one reverse proxy, the callers are server-to-server or a handful of admin devices, you can issue per-service certificates, and you can commit to rotation. Caddy or Nginx both do it in an afternoon.
• Use a mesh when: you run a real Kubernetes cluster with many workloads and want per-pod identity and automatic rotation. Let the cluster earn its keep first, then add the identity layer.
• Use an overlay VPN when: humans, mobile apps, and heterogeneous devices are the clients. Authenticated encrypted transport without per-app certificates beats strict mTLS that nothing can use.
• Consider neither when: the service only binds loopback or your network is already isolated from untrusted hosts. mTLS adds a PKI you now operate, and a PKI with no distribution or rotation plan is a liability, not a control.

Official sources

• Caddy tls directive and client_auth: https://caddyserver.com/docs/caddyfile/directives/tls
• nginx ngx_http_ssl_module (ssl_client_certificate, ssl_verify_client, ssl_crl): https://nginx.org/en/docs/http/ngx_http_ssl_module.html
• TLS 1.3, RFC 8446: https://www.rfc-editor.org/rfc/rfc8446
• Access your homelab anywhere with a YubiKey and mTLS (smallstep): https://smallstep.com/blog/access-your-homelab-anywhere/
• Accessing homelab services remotely with mTLS: https://brandonrozek.com/blog/accessing-homelab-services-mtls/
• Network mTLS setup with Caddy and Nginx (apalrd): https://www.apalrd.net/posts/2024/network_mtls/
• Linkerd mutual TLS: https://linkerd.io/
• Istio PeerAuthentication: https://istio.io/latest/docs/reference/config/security/peer_authentication/
• Our reverse proxy comparison: https://systhoughts.com/posts/caddy-vs-traefik-vs-nginx-proxy-manager
• Our Fail2ban vs CrowdSec post: https://systhoughts.com/posts/fail2ban-vs-crowdsec
• Our NetBird vs Pangolin post: https://systhoughts.com/posts/netbird-vs-pangolin-newt
• Our Docker secrets post: https://systhoughts.com/posts/docker-secrets-arent-really-secrets
• Our Compose vs Kubernetes post: https://systhoughts.com/posts/docker-compose-vs-kubernetes-self-hosted-apps

Are you running mTLS on internal services, or did you bounce off the mobile clients and the certificate renewal? Where did the line between useful security and theatre land for you? Drop it in the comments.

Until next time, keep your systems thoughtful.