AIDE and inotify-tools: File Integrity Baselines and Real-Time Filesystem Events, Hands-On

Every box I manage eventually asks the same two questions, and they sound alike but are not the same. Question one: has any file on this machine changed behind my back since I last looked? Question two: I need to react the instant a file changes, how? AIDE answers the first question, inotify-tools answers the second, and a lot of sysadmin work is really just knowing which question you are asking.

AIDE (Advanced Intrusion Detection Environment) is a file and directory integrity checker that builds a baseline database of hashes and attributes, then compares the live filesystem against it on your schedule. inotify-tools is a set of command line utilities that expose the kernel's inotify interface to shell scripts, so you can block on an event or count events as they happen. Both are free, both are GPL, both are in every distro repository, and neither needs a database server, an agent, or a cloud account.

This post is the hands-on version I wish I had found years ago: what each tool actually is, how you install and configure it on Debian, Ubuntu, and RHEL, how you wire AIDE into a daily check and inotify into a watcher script, the limits nobody mentions until they bite, and the honest trade-offs between a baseline you review on a schedule and a trigger that fires in milliseconds.

The short version

• AIDE is a file integrity checker, not a real-time watcher. It builds a baseline database from regex rules in a plain text config, then --check compares the filesystem against it. New files, removed files, and changed files are reported with their exit codes.
• inotify-tools is a real-time event toolkit. inotifywait blocks until a watched path produces an event, and inotifywatch counts events over a window and prints a table. Both work from shell scripts.
• AIDE answers "what changed since the last check?" and is perfect for tamper evidence, compliance, and drift checks on your schedule.
• inotify-tools answers "what is changing right now?" and is perfect for automation triggers, hot reloads, and reacting to uploads in milliseconds.
• The honest limits are the mirror image of each other. AIDE misses everything that happens between runs; inotify only watches the paths you explicitly watch, cannot watch network filesystems, and can drop events when the kernel queue overflows.

What AIDE actually is

AIDE has been around since 1999. It was originally written by Rami Lehti and Pablo Virolainen, and it has been maintained by Hannes von Haugwitz since October 2010. The design is deliberately boring: you write rules in a plain text config, each rule maps a regular expression over file paths to a set of attributes, and AIDE records those attributes in a baseline database. When you run a check, it re-reads the files, computes the attributes again, and prints every difference.

The database stores message digests plus the usual file metadata. Supported digests include md5, sha1, rmd160, tiger, crc32, sha256, sha512, and whirlpool, with more available if built with libmhash. The attribute set covers file type, permissions, inode, uid, gid, link name, size, block count, number of links, mtime, ctime, atime, and, when compiled in, POSIX ACLs, SELinux contexts, XAttrs, and extended filesystem attributes. That is the important part: AIDE is not just a checksum store, it notices a chmod, a chown, or a touched timestamp even when the content is untouched.

On Debian and Ubuntu the config lives at /etc/aide/aide.conf, which pulls in snippets from /etc/aide/aide.conf.d, and the database lives at /var/lib/aide/aide.db. On RHEL and Fedora the config is /etc/aide.conf and the database is the gzipped /var/lib/aide/aide.db.gz. The commands are identical everywhere.

The daily workflow is: --init once to create the baseline, then --check on a schedule. --update is the dangerous cousin: it checks, then accepts the current state as the new baseline. --compare compares two databases, which is how you diff your live baseline against the stored copy you keep off the host. Since version 0.18, AIDE can use multiple worker threads for hashing (num_workers in the config, --workers on the command line), which makes big-tree checks noticeably faster on multicore boxes.

Hands-on: initialize AIDE on Debian, Ubuntu, and RHEL

The first run is the only one that needs thought, because everything after it is compared against what you record now.

$ sudo apt install aide

# Debian ships aideinit, which wraps aide --init and places the database
$ sudo aideinit
# creates /var/lib/aide/aide.db.new

# RHEL family instead:
# sudo yum install aide
# sudo aide --init
# sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

# baseline is live; run a manual check to prove it works
$ sudo aide --check
# exit status: 0 = no differences; 1 = new files; 2 = removed; 4 = changed
# 3 = new and removed, 7 = all three, and so on

Those exit codes are the sysadmin-friendly part. Because they are additive bit flags, a script can decide what to do: ignore 1 (new files might be normal), page on 4 (a changed file is suspicious), and treat anything nonzero as something to log. Most AIDE integrations in configuration management read exactly this status.

The Debian default config takes a very paranoid stance, which is a polite way of saying it checks the entire filesystem including /home, and it will report a lot of noise on a box with active users. The fix is not to delete rules, it is to exclude what you do not care about. Rules are written as path regex plus an attribute group, and a leading ! excludes. A minimal custom section in /etc/aide/aide.conf.d/10-local.conf looks like this:

# a compact custom group: permissions, type, inode, links, uid, gid,
# size, mtime, ctime, sha256, plus xattrs/ACL when compiled in
Custom = p+ftype+i+n+u+g+s+m+c+sha256+X

/etc/passwd$ Custom
/etc/shadow$ Custom
/etc/ssh/sshd_config$ Custom
/boot$ Custom

# keep the noise out
!/home/activeuser
!/var/log
!/var/tmp

p is permissions, i is inode, n is link count, u and g are uid and gid, s is size, m and c are mtime and ctime, and X is the group that expands to ACL plus SELinux plus xattrs plus capabilities when available. The shipped config already defines useful groups: R is the recommended standard set with sha3_256, L is a lighter set, > is for growing logfiles, and H is every compiled-in hash. You can use those directly instead of writing your own.

Scheduling the daily check

On Debian, the aide package does the boring work for you. Since aide 0.18 the daily job runs as a dedicated non-root user _aide with the cap_dac_read_search capability, which lets it read anywhere without being root. On a systemd machine it runs as a system timer between 02:00 and 04:00, and on a non-systemd machine it falls back to /etc/cron.daily/aide.

# Debian runs aide as a systemd timer between 02:00 and 04:00,
# falling back to cron.daily when systemd is not PID 1
$ systemctl list-timers 'aide*'

# behavior is controlled by /etc/default/aide
$ grep -E 'COMMAND|COPYNEWDB|MAILTO|QUIETREPORTS' /etc/default/aide

# the daily report lands here
$ sudo tail -n 40 /var/log/aide/aide.log

/etc/default/aide is where the real policy lives. COMMAND decides whether the timer checks or updates, COPYNEWDB controls whether a fresh database is copied over the reference, MAILTO decides who gets the report, and QUIETREPORTS=yes suppresses the all-clear mail. The default COPYNEWDB=no is the safe choice: every change is re-reported until you accept it, which is annoying but honest.

On RHEL and Fedora there is no default timer, so you write your own systemd timer or cron entry that runs aide --check and logs the exit status. The pattern is identical to any other daily job, and since AIDE ships a static-ish binary and needs no daemon, a simple OnCalendar=*-*-* 03:00:00 timer is plenty.

What inotify-tools actually is

The inotify API was merged into the Linux kernel in 2.6.13, which shipped in 2005. It lets a process ask the kernel to watch files and directories and deliver events through a file descriptor: create, modify, attrib, closewrite, open, access, movedto, moved_from, delete, and friends. Unlike AIDE, there is no baseline and no stored state. The kernel is the watcher, and your process reads events as they happen.

inotify-tools is the package that makes that API usable from shell scripts. inotifywait blocks until a matching event occurs (or keeps monitoring with -m), and inotifywatch samples events for a set number of seconds, then prints a per-file event count table.

Newer releases also ship fsnotifywait and fsnotifywatch, which use the fanotify interface instead (event reporting since kernel 5.9, and admin privileges required since 5.12), plus a whole-filesystem mode with -S. The upstream README now recommends contributing to the Rust rewrite of the codebase rather than the C version, but the command line interface is what matters here and it is stable.

Hands-on: watch a directory with inotifywait

Install, then point it at a path and see what happens.

$ sudo apt install inotify-tools

# one-shot: exit after the first event
$ inotifywait /srv/uploads

# monitor mode, recursive, filter events
$ inotifywait -m -r -e create -e moved_to /srv/uploads

# machine-friendly output for scripts
$ inotifywait -m -r -e close_write \
    --format '%w%f %e' /srv/uploads

# exit codes: 0 = event seen, 1 = watch removed/error, 2 = timeout (-t)

The flags that matter: -m keeps it running instead of exiting after one event, -r recurses into subdirectories (one watch per directory, so the limits section below matters), -e selects the events you care about, and --format '%w%f %e' gives you machine-readable output with the watched path, the filename, and the event names. --exclude and --excludei take a POSIX regex to skip files you do not care about.

A practical upload watcher that processes each completed file once looks like this:

#!/usr/bin/env bash
# process every file that lands in /srv/uploads, once, when it is complete
inotifywait -m -r -e close_write -e moved_to --format '%w%f' /srv/uploads |
while read -r path; do
  case "$path" in
    *.part|*.tmp) continue ;;   # still being written
  esac
  echo "$(date -Is) processing $path"
  ./handle-upload.sh "$path"
done

Run that as a systemd service with Restart=always and the usual hardening (no new privileges, read-only root), and you have a mini event-driven pipeline that never polls.

Count events with inotifywatch

When the question is not "which file changed" but "how much activity is this tree generating", inotifywatch is the tool. Give it a duration and it prints a table of event counts per file or directory.

$ sudo apt install inotify-tools

# sample for 60 seconds, then print a per-file event count table
$ inotifywatch -t 60 -e access -e modify -e attrib -r /var/log

# include zero rows/columns, sort by total
$ inotifywatch -t 60 -z -a total -r /var/log

# fanotify variants (kernel 5.9+, admin privileges since 5.12)
$ fsnotifywatch -t 30 -S /home

This is a great way to find out which directory a noisy application is hammering before you tune it, or to prove to a colleague that a service is writing to a log far more often than it should. The output is one row per watched path and one column per event type, which is trivially greppable.

The limits that bite

Both tools have sharp edges, and knowing them is half the value.

• AIDE only sees what is in its rules and only when it runs. A change made and reverted between two daily checks is invisible. That is by design, but it means AIDE is evidence, not real-time protection.
• inotify cannot watch network filesystems. NFS and friends do not deliver inotify events, which is why the man page says remote filesystems are out of scope. Fanotify has the same constraint in practice.
• The inotify event queue can overflow. Each inotify instance has a queue capped by /proc/sys/fs/inotify/max_queued_events (16384 by default on most distros). If a burst outruns your consumer, events are dropped and an IN_Q_OVERFLOW event is emitted instead. This is why inotify is a trigger, not an audit log, and why the kernel documentation explicitly says you cannot reliably count events with it.
• Recursive watches are limited per user. -r creates one watch per directory. Raise it, if necessary, with e.g. sysctl fs.inotify.max_user_watches=524288 (and the matching max_user_instances) if you watch big trees.
• There are races in recursive watching. The man page is honest about this: events that occur in a directory immediately after that directory is created can be missed. Combine inotify with a periodic AIDE check and the holes mostly cancel out.

AIDE vs inotify-tools at a glance

Using them together

They are complementary, and the strongest setups run both. AIDE owns the baseline: it tells you, daily, that /etc/passwd, /etc/ssh/sshd_config, and your binaries have not drifted, and it gives you an exit code you can feed into monitoring. inotify-tools owns the moment: it tells you, instantly, when a file lands in the upload directory, when a config file is edited, or when something writes to a directory it should not.

The pattern I use on my own boxes: AIDE on a systemd timer for the daily evidence, and an inotifywait service on sensitive directories that logs the event and fires an alert when a path I did not expect changes. That division matches the defense-in-depth idea we covered in the Fail2ban vs CrowdSec post, where Fail2ban handles the network layer and the integrity layer should be a separate tool that does not depend on it. And because both tools are small, local, and version-stable, keeping them current is just part of the ordinary release tracking workflow rather than a project of its own.

Which should you pick?

• Choose AIDE when you need tamper evidence, compliance reporting, or drift detection across a whole tree on your schedule. The baseline database and the additive exit codes make it easy to prove that nothing changed, which is exactly what audits want.
• Choose inotify-tools when you need to react the instant a file changes: process uploads, hot-reload a service, trigger a backup, or alert on writes to a sensitive path. It is the right tool for automation, and it is why so many tools internally use inotify.
• Run both when you want instant reaction plus a durable baseline. They do not overlap, and the holes in one are covered by the other.
• Consider neither when you need syscall-level audit records with unskippable provenance, which is the job of auditd, osquery, or a proper SIEM-grade agent. AIDE proves content and metadata integrity; it does not record the history of every process that touched a file.

Official sources

• AIDE: https://aide.github.io/
• aide(1): https://manpages.debian.org/trixie/aide/aide.1.en.html
• aide.conf(5): https://manpages.debian.org/trixie/aide/aide.conf.5.en.html
• AIDE for Debian (packaging README): https://github.com/ossobv/aide-deb
• RHEL security hardening, checking integrity with AIDE: https://docs.redhat.com/en/documentation/redhatenterpriselinux/8/html/securityhardening/checking-integrity-with-aide_security-hardening
• inotify-tools: https://github.com/inotify-tools/inotify-tools
• inotifywait(1): https://man7.org/linux/man-pages/man1/inotifywait.1.html
• inotifywatch(1): https://man7.org/linux/man-pages/man1/inotifywatch.1.html
• inotify(7): https://man7.org/linux/man-pages/man7/inotify.7.html

Are you running AIDE, inotify-tools, or both on your boxes? Which directory did you point inotifywait at first, and what did it catch that you did not expect? Drop it in the comments.

Until next time, keep your systems thoughtful.