Docker Compose vs Kubernetes for Self-Hosted Apps: When the Cluster Earns Its Keep

If you run more than a handful of self-hosted services, you eventually stare at your Compose files and ask the question every sysadmin asks at some point: "should I move this to Kubernetes?" The last post in this series covered Docker vs Podman for self-hosted apps and ended with a one-line answer: Kubernetes is a platform, not a container runtime, and if you are running one server, it is a second job, not an upgrade. This post is the longer version of that answer, aimed at the sysadmin who runs many apps on hardware they control and wants to know when the cluster actually earns its keep.

I cover what each tool actually is, where Compose starts to hurt at many apps, the honest cost of running Kubernetes at home, and the thing every comparison glosses over: rootful on Kubernetes. A default cluster runs a rootful kubelet and rootful containerd on every node, so the security story is not "rootless like Podman". It is a rootful platform defended by multiple layers: RBAC, admission control, pod securityContext, seccomp, and network policies. Understanding how those layers work, and how they differ from the "docker group is root" model, is the difference between a cluster that protects you and one that just adds a control plane to your problems.

The short version

• Compose is a single-host orchestrator; Kubernetes is a platform. Compose declares multi-container stacks for one host. Kubernetes reconciles desired state across a cluster with a control plane. "Compose vs Kubernetes" is really "do I need a scheduler and a control plane yet?"
• Both run rootful runtimes by default, and the difference is structural. Docker concentrates power in one root daemon with a socket (docker group ~ root). Kubernetes runs rootful kubelet and containerd but exposes no user socket: everything goes through the API server with TLS, RBAC, and admission control.
• Kubernetes hardens containers with layers Docker leaves to you. Pod Security Standards, admission policies, securityContext, seccomp profiles, network policies, and per-namespace RBAC. The platform is rootful by default; the defense is the layers, not rootlessness.
• k3s is the self-hoster's on-ramp. One binary around 100 MB, bundled containerd, Traefik ingress, local-path storage, SQLite instead of etcd. It runs on the same N100 that runs your Compose stacks.
• Kubernetes earns its keep when a second host exists. Node failure reschedules, rolling updates gate on health, namespaces give other people an island. On one box, it is a lot of machinery for the same failure domain. It gets real on 9+ nodes.
• Compose is not a failure state. If you never plan a second host, the most documented path in existence is still the right tool. Kubernetes is an option you graduate into, not a mandatory step.

What each tool actually is

Compose: a single-host orchestrator in one YAML file

Docker Compose is the de facto standard for multi-container stacks on one host, and it inherits the architecture of Docker Engine: a root dockerd daemon, containerd underneath it, and a socket whose access is effectively root. The Docker vs Podman post covered that rootful story in depth, including the honest limits of rootless alternatives, so I will not repeat it all here; the short version is that Compose gives you stack-level management, portability, and the most documented path in existence, and its security model is "the docker group is root, so lock down the socket."

What Compose does not do is schedule across hosts. Every service is pinned to the machine where you ran docker compose up -d. There is no scheduler that moves a workload when the host dies, no rolling update that gates on health checks, no RBAC, no admission control, no network policy. Those are not bugs; they are the boundaries of the tool, and they are exactly the features Kubernetes adds.

Kubernetes: a platform, not a container runtime

Kubernetes is a control plane plus a set of nodes. The control plane (API server, etcd or Kine, scheduler, controller managers) holds desired state and drives the system toward it. On each node, the kubelet talks to the API server and supervises the container runtime, which is containerd (or CRI-O), the same runtime that powers Docker. A Deployment declares "I want 2 replicas of this pod, always", and controllers turn that into running containers, restarting them on crash, rescheduling them on node failure, and rolling them out on image change.

The mental model shift is the important part: with Compose you run containers; with Kubernetes you declare state and the platform reconciles it. That is why people call it a platform rather than a container runtime. It is also why the honest comparison is not "Compose vs a runtime", it is "a single-host orchestrator vs a multi-host platform".

Where Compose starts to hurt at many apps

For ten or fifteen services on one N100, Compose is genuinely fine. The pain shows up in specific places:

• No scheduler. Every service is pinned to the host it started on. Moving a service to another machine means docker compose up -d somewhere else and manual port reconfiguration. There is no "put this somewhere with free RAM."
• One host, one failure domain. Restart policies survive a daemon restart, not a host failure. If the box dies, the stack dies with it, and recovery is you, at a console, at 2 AM.
• No health-gated deploys. docker compose up -d recreates containers in sequence. If the new image is broken, the service is down until you revert. There is no rollout that pauses because the health check fails.
• No RBAC or admission control. Anyone with the docker socket or group has root. There is no "namespace for my friend's side project" and nothing stops a container from being launched privileged by accident.
• No network policy. On the default bridge network, any container can reach any other. Isolation is hand-rolled firewall rules.
• Secrets are env files. No encryption at rest, no rotation story, no audit trail. (Kubernetes fixes some of this; see the callout about base64 below, because it only fixes part of it.)

None of this means Compose is bad. It means the pain is concentrated at the boundaries: multiple hosts, other people, and deploys you cannot afford to take down.

Rootful on Kubernetes: what runs as root and how it is defended

This is the section most comparisons skip, and it is the one that actually matters for the security decision. Let us be precise.

What is actually rootful

On a default install, including a default k3s install: the kubelet runs as root, containerd (or CRI-O) runs as root, the CNI plugin runs as root, and kube-proxy runs as root. The kubelet needs root because it mounts volumes, sets up cgroups, manages the pod sandbox, and applies the pod's security context. So yes: Kubernetes is rootful by default, and it is not Podman's rootless model. Anyone who tells you "Kubernetes is secure because it is rootless" is wrong.

The key difference from Docker: there is no socket-to-root path

The difference from Docker is not rootless vs rootful. It is where the power lives and how you get to it.

With Docker, the juiciest target on the host is one root daemon with an API, and the path to it is short: the socket, or membership in the docker group. A compromised container that can reach the socket is root, full stop. The mitigation ladder (don't expose the socket, don't add users to the group, drop capabilities) is per-host discipline with no enforcement.

Kubernetes removes that path by construction. There is no user-facing root socket. The kubelet does expose a local API on port 10250, but it authenticates and authorizes requests, and anonymous auth is disabled by default. Humans and workloads reach the cluster through the API server over TLS, and the API server is the only door. That door is defended by three gates in series: authentication (who are you: client certs, tokens, OIDC), authorization (what may you do: RBAC), and admission control (is this object allowed: Pod Security Standards, policy engines, image policies). There is no equivalent of "add yourself to the docker group."

That is the structural headline of this post: Docker concentrates power in a root daemon and guards it with group membership; Kubernetes keeps a rootful runtime but routes every access path through policy. Rootful is the shared starting point; accountability is where they diverge.

Layer 1: the control plane gate

• The API server speaks TLS only. A kubeconfig carries a client certificate or token; nothing listens on an unauthenticated socket.
• RBAC grants least privilege per namespace. Users, groups, and service accounts get roles; there is no implicit "member of the group is root."
• The node authorizer and NodeRestriction admission plugin limit what a kubelet can do: it can modify its own node and the pods bound to it, and nothing else. That constraint matters in the escape scenario below.
• Service account tokens are projected, short-lived, and audience-bound on modern Kubernetes, instead of long-lived values sitting in a Secret.

Layer 2: admission control

Admission runs before an object is persisted, which is the difference between "you should not do this" and "you cannot do this."

• Pod Security Standards (baseline and restricted) via Pod Security Admission labels on a namespace. The restricted profile forbids privileged containers, hostPath mounts, hostNetwork, and hostPID, and requires runAsNonRoot and a seccomp profile. Namespace labels make it default-on for everything that lands there.
• Policy engines (Kyverno, OPA Gatekeeper) can go further: require image digests, enforce resource limits, add defaults to every pod.
• Image policy: sign images with cosign and verify at admission, so only signed images reach the cluster.

Layer 3: the pod securityContext

The pod-level hardening that Docker gives you as run flags becomes declarative and enforceable:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vaultwarden
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vaultwarden
  template:
    metadata:
      labels:
        app: vaultwarden
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: vaultwarden
          image: vaultwarden/server:latest
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              memory: 128Mi
            limits:
              memory: 1Gi

Every field here maps to a flag you could pass to docker run: --user, --security-opt seccomp=..., --cap-drop ALL, --read-only, --memory. The difference is that in Kubernetes this is part of the workload declaration, it can be defaulted by admission for every pod in a namespace, and a change is a reviewable diff, not a flag you hope you remembered.

Layer 4: kernel hardening

• seccomp: modern kubelets (and k3s) can default to the RuntimeDefault profile, and you can pin per-pod profiles for the paranoid workloads.
• AppArmor and SELinux: profiles attach to pods; on Fedora or similar, SELinux contexts apply per container.
• User namespaces for pods (stable as of Kubernetes 1.36 but still opt-in via hostUsers: false, with rootless containerd as the runtime-side complement) are the only real "rootless pod" path. The production default remains rootful plus the layers above.

Layer 5: network and data

• NetworkPolicies give you default-deny between pods. One caveat for k3s users: the bundled Flannel CNI does not enforce policies by default. You need a CNI with policy support (Calico, Cilium, or kube-router) if you want them to bite.
• Secrets: base64 only, unless you enable encryption at rest for the API data plane. See the callout below; this is the most common false sense of security in Kubernetes.
• TLS everywhere: Ingress terminates TLS at the edge; etcd peer and client TLS should be on (k3s enables etcd TLS out of the box).

The escape scenario, and why the layers matter

Walk the worst case: a container is compromised, an exploit escapes the namespaces, and the attacker gets root on the node. What happens next is where Kubernetes differs from Docker.

In Docker, "root on the node" is the end of the game. The attacker owns the host and everything on it. In Kubernetes, root on the node gets you the kubelet's credentials, and the node authorizer and NodeRestriction plugin mean those credentials can only touch the node you escaped and the pods bound to it. You cannot create pods in other namespaces, you cannot read every Secret, you cannot drain the cluster. Pivoting to the API server requires defeating the RBAC gate with an identity that was deliberately starved of power.

That is the honest win: the blast radius of a node compromise is capped by policy, not by luck. It is not magic. If your cluster-admin role is everywhere, if every namespace is unrestricted, if secrets are readable cluster-wide, then an escape is still a disaster. The layers only protect you if you configure them, and the default posture of most distros is far from locked down. Harden before you add workloads, not after.

k3s: the self-hoster's on-ramp

If the sections above convinced you the model is worth trying, k3s is the way to do it without renting a control plane. It is a CNCF-certified Kubernetes distribution in a single binary of around 100 MB: bundled containerd, Flannel, CoreDNS, Traefik as the default ingress, a local-path provisioner for storage, and SQLite (via Kine) instead of etcd unless you opt into etcd.

$ curl -sfL https://get.k3s.io | sh -
# kubeconfig lands at /etc/rancher/k3s/k3s.yaml
$ sudo k3s kubectl get nodes
NAME    STATUS   ROLES                  AGE   VERSION
n100    Ready    control-plane,master   42s   v1.36.3+k3s1

$ export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
$ kubectl create deployment vaultwarden --image=vaultwarden/server:latest
$ kubectl rollout status deployment/vaultwarden
$ kubectl get pods -o wide

It runs happily on the same N100 that runs your Compose stacks, which is both the appeal and the trap. Single-node k3s gives you the full API and workload model: Deployments, Services, Ingress, namespaces, RBAC, rollouts. What it does not give you is a second failure domain. If the node dies, the cluster dies, exactly like Compose on the same hardware, just with more machinery to restart. Treat single-node k3s as a learning platform and a stepping stone to a second host, not as high availability.

Upgrades, restarts, and day-2 operations

The operational rhythm changes more than the config files do:

• Compose: edit the YAML, docker compose pull && docker compose up -d, rollback is a git revert (or a image pinned) and the same command. One file, one command, whole stack moves together.
• Kubernetes: kubectl rollout restart deployment/name or helm upgrade; kubectl rollout status and kubectl rollout undo give you health-gated, reversible deploys. k3s itself upgrades through a systemd unit swap or the system-upgrade-controller, so the cluster is a rolling component you now maintain alongside your workloads.
• Watchtower has no clean Kubernetes equivalent. The closest patterns are image pull policies plus a rollout restart, or moving to GitOps (Flux or Argo CD), where the git repo is the desired state and the operator applies it. That is a real workflow change, not a cosmetic one.

The control plane is a thing you now own: kubeconfig and certs (k3s auto-renews its own), etcd or Kine backups, CNI upgrades, ingress controller upgrades. For a homelab this is an hour a month of maintenance; for a production box it is a job.

Resource limits and the rest of the comparison

The honest cost of running Kubernetes at home

The features are real; so is the price. A single-node k3s box is more moving parts than a Compose box for the same failure domain, and the platform itself becomes a thing to upgrade, back up, and debug. The control plane, the CNI, the ingress controller, and the storage provisioner are now your responsibility, and they fail in new and interesting ways that Compose never will.

The moment it flips is a second host. Add one more machine, and Kubernetes stops being overhead and starts being the thing that keeps your services up when a node dies, that rolls updates across both boxes, and that gives you namespaces so the family's media stack cannot touch your production automations. Add a third and it is the only sane way to run the fleet.

For the specific case in the Docker vs Podman post, nothing changes: my n8n box runs Compose, and it should. It is one machine, the workload is a stack, and the ecosystem is Docker-first. The day I add a second server, the conversation starts, and it starts with k3s.

Which should you pick?

Docker Compose when: you run one host; your services are stacks that deploy as a unit; you want the file to be portable across machines, CI, and other people's setups; your ecosystem assumes the Docker API; the box is single-tenant and you lock down the socket. This remains the most documented path in existence, and it is the right default for the overwhelming majority of self-hosters.

Kubernetes when: you have two or more hosts and want node-failure rescheduling and rolling updates with health gates; you need namespaces and RBAC so other people can have their own islands; you want declarative, GitOps-style state instead of imperative commands; or you are learning it deliberately and can absorb the maintenance tax. Start with k3s, keep the kubeconfig backed up, and only call it a cluster when the second node exists.

And the honest middle, same as the Podman post: a plain systemd service from the distro package is still the lowest-maintenance option for software that does not need the isolation, and Compose is still the lowest-maintenance option for a stack on one box. Kubernetes is a platform you graduate into when the problem outgrows the box, not a badge of honor to wear early. For serious, realiable HA we’re talking about 9+ nodes.

What is running your stack: Compose, a k3s cluster, or a mix? Drop it in the comments.

Until next time, keep your systems thoughtful.