Linux Watchdog: What It Is, When It Helps, and How to Set It Up

If you have been following the self-hosting thread on this blog, you know the pattern: every layer of the stack gets asked the same question. The backup post asked what survives ransomware, Fail2ban vs CrowdSec asked who is allowed to talk to the box at all, and the AIDE and inotify-tools post asked whether the files on it have quietly changed.

None of them answered the question that starts with a machine that has simply stopped responding: if the kernel is too broken to notice that it is broken, what reboots the server?

That is the job of the watchdog timer, and it is one of the few parts of the stack that predates Linux: a countdown circuit that resets the machine when nobody pings it in time. The Compose vs Kubernetes post framed the container decision as choosing which failure mode you would rather debug. A watchdog makes the same kind of choice, with a blunt edge: a reboot instead of silence.

This post is the sysadmin walkthrough. What the kernel watchdog actually is, why a hardware watchdog and the softdog module are not the same thing, how systemd, a systemd unit, and the userspace watchdog daemon each sit on top of it, and five steps to arm it on your own box without bricking it.

The short version

• A watchdog is a countdown timer with a reboot attached. Userspace pings it, usually by writing to /dev/watchdog, the kernel resets the countdown on every ping, and if the pings stop the machine resets. That is the entire mechanism.
• Hardware watchdogs survive a wedged kernel; softdog usually does not. The timer chip keeps counting even if Linux is deadlocked, so it can still reset the box. A software watchdog needs the kernel's own timer subsystem to be working, which is exactly what a hard hang breaks.
• systemd already has the plumbing, and it is off by default. RuntimeWatchdogSec= in /etc/systemd/system.conf.d/ arms the hardware watchdog and pings it for you. Defaults: RuntimeWatchdogSec=0 (off), RebootWatchdogSec=10min, WatchdogDevice=/dev/watchdog0.
• `WatchdogSec=` in a unit is a different layer. That one watches a service, not the machine: the service has to send keep-alives over the notify socket, and systemd restarts it when they stop.
• One device, one owner. The userspace watchdog daemon and systemd cannot both hold /dev/watchdog. Pick one and turn the other off.
• Test with softdog's `soft_noboot=1` or the daemon's `-q` flag before you trust it. The first test of a watchdog on a box you care about should not involve a real reset.

What a watchdog actually is

A watchdog timer is a hardware circuit that can reset the computer if software stops proving it is alive. Linux exposes those timers as character devices: /dev/watchdog0, /dev/watchdog1, and so on, with /dev/watchdog kept as a compatibility alias for the first one. Each device also shows up under /sys/class/watchdog/watchdogN/ with its identity, current timeout, whether it is armed, and whether it can be stopped at all.

The contract is small, and every part of it has bitten somebody:

• Opening the device arms the timer. Most drivers start counting as soon as the device is opened, so a bug in your watchdog daemon is an outage, not just a failed service.
• Writing to it, or the `WDIOC_KEEPALIVE` ioctl, resets the countdown. That is the ping. Some drivers also accept WDIOC_SETTIMEOUT to change the timeout at runtime, and they report back the timeout actually used, which can be coarser than what you asked for because of hardware granularity.
• Closing the device disarms it, unless the driver supports Magic Close. With Magic Close, the driver only disarms the timer if you write the character V immediately before closing. Close without it and the driver assumes userspace died, leaves the timer running, and the box reboots if nothing reopens the device in time.
• `nowayout` removes the off switch. With CONFIG_WATCHDOG_NOWAYOUT or nowayout=1, the watchdog cannot be stopped once started. If your watchdog daemon crashes, the machine reboots, which is usually the point, and it also means a broken watchdog daemon becomes a reboot loop.
• Some drivers can warn before they reset. A pre-timeout fires first, and the kernel can be told to take an action such as a panic when it does. That is how you get a crash dump instead of a silent reset.

A minimal software watchdog is about ten lines of C, which is worth knowing: nothing here is magic, and any component that can ping the device can be the thing keeping the box alive.

Hardware versus software: pick the one that matches your failure mode

The distinction that matters operationally is whether the timer depends on the operating system. A chipset or BMC timer is independent of Linux, so a deadlocked kernel, a stuck driver, or even a halted CPU will not stop it from firing. softdog is a kernel module that emulates the same interface with a kernel timer, which means it can cover a hung userspace, a stopped daemon, or a process stuck in uninterruptible sleep, and it cannot cover the kernel scheduling itself to death. The watchdog man page says the quiet part out loud: with the software watchdog, whether you get a reboot depends on the state of the machine and interrupts.

So the honest rule: if your box has a watchdog chip, use it. If it does not, softdog still buys you recovery from one class of hangs, and it does not protect you from the hang that most looks like it needs a hardware solution.

Where it fits in the stack

Four layers, and they are not replacements for each other:

The first two are about recovering a machine that has stopped running. The third restarts a sick service. The fourth is the only layer that can decide the machine is unhealthy before it stops responding at all.

Step 1: find out what the box actually has

Before you configure anything, learn whether there is a timer at all, which driver owns it, and whether it is currently armed.

$ ls -l /dev/watchdog*
lrwxrwxrwx 1 root root 0 Sep 10 09:12 /dev/watchdog -> watchdog0
crw------- 1 root root 10, 130 Sep 10 09:12 /dev/watchdog0

$ ls /sys/class/watchdog/
watchdog0

$ cat /sys/class/watchdog/watchdog0/identity
iTCO_wdt
$ cat /sys/class/watchdog/watchdog0/timeout
60
$ cat /sys/class/watchdog/watchdog0/nowayout
0

$ dmesg | grep -i watchdog
[    2.118431] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[    2.118521] iTCO_wdt: Found a Intel PCH TCO device

$ wdctl
Device:        /dev/watchdog0
Identity:      iTCO_wdt [version 0]
Timeout:       60 seconds
Pre-timeout:    0 seconds
Timeleft:      59 seconds
FLAG            DESCRIPTION                  STATUS BOOT-STATUS
KEEPALIVEPING   Keep alive ping reply              1           0
MAGICCLOSE      Supports magic close char          0           0
SETTIMEOUT      Set timeout (in seconds)           0           0

Two caveats about wdctl. It reads from sysfs when the device is already in use or you do not have permissions, which means some flags go missing. And on drivers where nowayout is disabled, opening the device is enough to take ownership of the timer, so a casual run can leave the watchdog off with no error message. Check nowayout first, or run it when you are ready to re-arm.

Step 2: let systemd own the hardware watchdog

If a watchdog device exists, this is the whole setup. Use a drop-in for the manager rather than editing the shipped file, and choose a timeout that is comfortably longer than any legitimate pause (a slow resume, a heavy database checkpoint, a backup window) and comfortably shorter than your patience.

# /etc/systemd/system.conf.d/10-watchdog.conf
[Manager]
RuntimeWatchdogSec=30s
RuntimeWatchdogPreSec=10s
RuntimeWatchdogPreGovernor=panic
RebootWatchdogSec=2min
WatchdogDevice=/dev/watchdog0

Then re-execute the manager and verify, remembering that systemd pings at least once in half the configured interval, so a 30 second timeout gets a ping every 15 seconds at worst.

$ sudo systemctl daemon-reexec

$ systemctl show --property RuntimeWatchdogUSec --property RebootWatchdogUSec
RuntimeWatchdogUSec=30s
RebootWatchdogUSec=2min

# the device should now report a short timeout and a countdown
$ wdctl
Device:        /dev/watchdog0
Identity:      iTCO_wdt [version 0]
Timeout:       30 seconds
Timeleft:      29 seconds

Three details worth knowing:

• `daemon-reload` does not re-read manager configuration. The daemon-reexec call above is what applies these settings, and it is also how you apply a changed timeout later.
• `RebootWatchdogSec=` is the reboot net, not a second runtime timer. It arms the watchdog for the shutdown handoff, after services are stopped and systemd-shutdown has replaced PID 1. The default is 10 minutes, which is a long time to sit on a box that will not power down.
• The pre-timeout is the part that gives you evidence. Setting RuntimeWatchdogPreGovernor=panic, where the device and kernel support it, turns a silent hang into a panic that kdump can capture, so the next mystery reboot comes with a stack trace.

Step 3: no watchdog chip? load softdog

Plenty of small boxes and most virtual machines expose no hardware watchdog at all. Step 1 tells you: if /sys/class/watchdog/ is empty and dmesg | grep -i watchdog shows nothing, softdog is your only option on the software side. Load it in test mode first, so nothing reboots while you look at it.

$ sudo modprobe softdog soft_margin=60 soft_noboot=1

$ ls -l /dev/watchdog*
lrwxrwxrwx 1 root root 0 Sep 10 09:12 /dev/watchdog -> watchdog0
crw------- 1 root root 10, 130 Sep 10 09:12 /dev/watchdog0

$ cat /sys/class/watchdog/watchdog0/identity
softdog

# soft_noboot=1 arms the timer and ignores expiry:
# it is the only way to inspect a live watchdog without risking a reboot
$ wdctl
Device:        /dev/watchdog0
Identity:      softdog
Timeout:       60 seconds
Timeleft:      59 seconds

Make it persistent with two small files, one to load the module and one to pass the options:

# /etc/modules-load.d/watchdog.conf
softdog

# /etc/modprobe.d/softdog.conf
options softdog soft_margin=60

Reboot, confirm the module came back loaded, and only then consider dropping soft_noboot=1. The other softdog options are soft_panic and nowayout, which are useful if you want a panic instead of a silent reset, or a watchdog that cannot be disarmed. And keep soft_margin generous: 60 seconds is a reasonable floor, a tight value that fires during a backup is the classic own goal, and if systemd is managing the timer you should let it own the timeout instead of setting a second one in the module.

Step 4: watch a single service with WatchdogSec

The kernel watchdog reboots the machine. That is a heavy tool for nginx quietly returning 502s, so systemd has a second, separate watchdog for services:

# /etc/systemd/system/myapp.service.d/watchdog.conf
[Service]
Type=notify
NotifyAccess=main
WatchdogSec=30s
Restart=on-watchdog
RestartSec=5s
StartLimitIntervalSec=5min
StartLimitBurst=5

The service has to hold up its end. It must send WATCHDOG=1 over the notify socket faster than WatchdogSec, which means calling sd_notify() from the application or systemd-notify WATCHDOG=1 from a wrapper loop. If the keep-alives stop, systemd fails the unit, and with Restart=on-watchdog (or on-failure, or always) it restarts. Without a restart policy, the unit simply goes failed and stays there, which surprises people who expect the watchdog itself to restart things.

This is the layer that gets confused with the kernel watchdog, and the confusion matters. It cannot recover a hung kernel, and it does not need to: it recovers a hung process, which is by far the more common failure. StartLimitBurst and StartLimitAction= are the bridge between the two layers, so if you want a service that keeps failing to escalate into a reboot, that is the setting to reach for.

Step 5: real health checks with the watchdog daemon

Sometimes liveness is not the question. You want the box to do something when the load is absurd, memory is gone, a temperature sensor is climbing, a file stopped changing, a process died, or a network interface went quiet. That is the userspace watchdog daemon, one package on Debian and Ubuntu that ships its own systemd unit.

Its defaults live in /etc/default/watchdog, and the interesting line is watchdog_module="none": the package will not load a kernel module for you, so the driver has to be loaded already (Step 3) or you let systemd own it. Set run_watchdog=1, and note that the package also ships wd_keepalive, whose job is to keep pinging the device while the main daemon is stopped for maintenance.

A reasonable starting configuration:

# /etc/watchdog.conf
watchdog-device = /dev/watchdog
watchdog-timeout = 60
interval = 10
realtime = yes
priority = 1
max-load-1 = 16
min-memory = 262144
log-dir = /var/log/watchdog
# repair-binary = /usr/local/sbin/repair-my-thing
# test-binary   = /usr/local/sbin/test-my-thing

Reading that: interval = 10 pings every ten seconds, realtime = yes locks the daemon into memory so that being swapped out under load cannot trigger a false reboot (the most important line here on a busy box, and the man page is explicit that a swapped-out watchdog will reset the machine), max-load-1 = 16 reboots when the one minute load average crosses 16 (the 5 and 15 minute thresholds default to fractions of it), and min-memory is counted in 4 kB pages, so 262144 pages is 1 GiB of free memory. Drop realtime and priority if your kernel or setup will not allow them.

Anything more specific goes in /etc/watchdog.d/. Every executable there is discovered at startup and called with test, and on failure called again with repair and the failing exit code, which is how you get a check that can heal a service instead of rebooting the box. The return codes are documented and worth reading: 255 means reboot now, 254 means hard reset, and specific values report load, temperature, memory and file-change failures.

Run it in the foreground with no action first. That executes every check and logs the result without touching the device:

# foreground, verbose, no action: run the checks, touch nothing
$ sudo watchdog -q -F -v -c /etc/watchdog.conf

# or call a single check script the way the daemon will call it
$ sudo /etc/watchdog.d/my-check test
$ echo $?
0

Now the part most guides skip. The daemon's thresholds can disagree with reality, and a false positive here reboots a healthy machine. max-load-1 is the usual culprit: a nightly backup, a container image build, or a ZFS scrub can push a small box past any threshold tuned on a quiet afternoon. Set it high, watch the journal and /var/log/watchdog for a few weeks, and tighten it only once you know what normal looks like on that hardware.

Prove it works, on a box you can lose

A watchdog that has never fired is a hypothesis. The only honest test is to simulate a hard hang and watch the machine come back by itself, which is why this belongs on a scratch VM or a disposable host rather than the box holding your only copy of anything.

# on a scratch VM, with kernel.sysrq enabled, crash the kernel on purpose
$ echo c | sudo tee /proc/sysrq-trigger

# the box should reset on its own once the timer expires
# after it comes back, ask what happened in the previous boot
$ journalctl -b -1 -k | grep -i watchdog
[  812.401123] watchdog: watchdog0: watchdog did not stop!

# and ask the driver whether the last reset was its doing
$ cat /sys/class/watchdog/watchdog0/bootstatus
# a non zero value here means the last reset came from the watchdog

The kernel message is the receipt. watchdog: watchdog0: watchdog did not stop! in the previous boot tells you the timer expired and reset the machine, rather than the box rebooting for some other reason. Between that line and the boot status in sysfs you have both halves of the evidence, and both are worth checking the first time a real hang happens.

What breaks first

• Boot loops get worse, not better. If the real failure is a bad kernel, a corrupt initramfs, or a full root filesystem, the watchdog turns a single outage into a machine that reboots every 60 seconds forever. systemd 261 added MinimumUptimeSec=, defaulting to 15 seconds, specifically to slow those loops down so you can still reach a console, but the real fix is the boot problem.
• Storage and network hangs are the wrong target. If the box hung because an NFS mount died or a disk is failing, a reset is not a repair, and it can interrupt recovery that was making progress. Read the logs for a cause before you let the timer decide.
• Two openers means somebody loses. That includes wdctl, which can disarm a running timer on drivers where nowayout is off.
• Cloud instances usually have no timer at all. Most providers expose no watchdog device, so RuntimeWatchdogSec= is a no-op there. On your own KVM and QEMU hosts you can attach an i6300esb device, and remember the reset then restarts the guest, which is a useful but different guarantee.
• Containers are not the right boundary. A container has no /dev/watchdog unless you pass one in, and it should not be able to reset the host anyway. This is a host-level concern, which is correct: it is the recovery layer underneath the platform, not inside it. The Docker vs Podman post makes the same ownership argument for runtimes.
• It is not a monitor and not a backup. A watchdog does not notice that a service is returning 500s, does not notice deleted files, and does not protect data. It covers the case where nothing else can act, and it sits beside the integrity checks from the AIDE and inotify-tools post rather than replacing them.
• Timing is a tuning problem, not a one-off setting. The right soft_margin, WatchdogSec, or RuntimeWatchdogSec depends on what your box does at 02:00. Pick generous values, look at the logs, and revisit them when the workload changes.
• Keep versions pinned and tracked. Timer behavior and options change between kernel, systemd and driver releases, which is the same reason the release-tracking workflow exists. Read the notes before a distro upgrade moves you forward.

Which should you pick?

• A modern box with a watchdog chip and systemd: set RuntimeWatchdogSec= and RebootWatchdogSec=, add the pre-timeout panic governor where the device supports it, and leave the rest alone.
• A box with no chip: load softdog, let systemd ping it, and accept that you are covering userspace hangs rather than kernel death.
• A box you want checked beyond liveness: the userspace watchdog daemon, with realtime = yes, generous thresholds, and a /etc/watchdog.d test that repairs before it reboots.
• A service that gets stuck while the machine stays up: WatchdogSec= in the unit, and leave the kernel watchdog out of it.
• Nothing: if the failure you fear is a boot loop, or the machine is a laptop in the same room, or it is a container, skip the whole exercise. A watchdog answers exactly one question, and if you are not asking that question, it is another moving part to maintain.

My own setup, for what it is worth: the N100 that runs my self-hosted n8n automation stack has an Intel TCO watchdog, so systemd arms it with a 30 second timeout and a 10 second pre-timeout that panics, and systemd does all the pinging. Everything else I rely on is the layer above: the checks that notice a problem while the box is still answering, and the backups that make a bad reboot survivable.

Official sources

• The Linux kernel watchdog driver API: https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html
• Watchdog module parameters: https://www.kernel.org/doc/html/latest/watchdog/watchdog-parameters.html
• systemd system manager configuration, Hardware Watchdog section: https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html
• systemd service configuration, WatchdogSec=: https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html
• sdnotify and the WATCHDOG=1 keep-alive: https://www.freedesktop.org/software/systemd/man/latest/sdnotify.html
• watchdog(8), the userspace daemon: https://manpages.debian.org/trixie/watchdog/watchdog.8.en.html
• watchdog.conf(5): https://manpages.debian.org/trixie/watchdog/watchdog.conf.5.en.html
• wdctl(8): https://man7.org/linux/man-pages/man8/wdctl.8.html
• Our file integrity and filesystem monitoring post: https://systhoughts.com/posts/aide-and-inotify-tools-file-integrity-and-filesystem-monitoring
• Our Compose vs Kubernetes post: https://systhoughts.com/posts/docker-compose-vs-kubernetes-self-hosted-apps
• Our backup and recovery keys post: https://systhoughts.com/posts/3-2-1-backup-rule-not-enough-self-hosted-infrastructure
• Our release-tracking workflow: https://systhoughts.com/posts/tracking-software-releases-across-forges

Has a watchdog ever caught a hang that nothing else would have, or rebooted a healthy box at the worst possible moment? What timeout did you settle on, and did you keep it tight or generous? Drop it in the comments.

Until next time, keep your systems thoughtful.