Issue#

On a busy Linux host or Kubernetes node, tools that watch files start failing even though ulimit -n is high and there are plenty of free file descriptors.

kubectl logs -f stops streaming:

failed to create fsnotify watcher: too many open files

tail -f falls back to polling:

tail: inotify cannot be used, reverting to polling: Too many open files

Other services such as log shippers, systemd, or IDE file watchers may report errors like:

inotify_init1() failed: Too many open files

Cause#

The message is misleading. The open file descriptor limit is not the problem. The kernel limits inotify instances per user through fs.inotify.max_user_instances, which defaults to 128 on many distributions. Every container, log agent, and kubectl logs -f session that watches files uses inotify instances. On a node running many pods, all of them owned by root, that limit is reached quickly.

A related limit, fs.inotify.max_user_watches, caps the total number of watched files. When that limit is reached, you usually see No space left on device instead.

Solution#

  1. Check the current limits:
sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches
  1. Find which processes are holding inotify instances:
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null \ | cut -d/ -f3 | sort | uniq -c | sort -nr | head \ | while read count pid; do echo "$count $(ps -o comm= -p "$pid")" done
  1. Raise the limits immediately. This change does not survive a reboot:
sudo sysctl -w fs.inotify.max_user_instances=8192 sudo sysctl -w fs.inotify.max_user_watches=524288
  1. Make the change persistent:
cat <<EOF | sudo tee /etc/sysctl.d/99-inotify.conf fs.inotify.max_user_instances = 8192 fs.inotify.max_user_watches = 524288 EOF sudo sysctl --system

  1. On a Kubernetes cluster, apply the same change on every node. For kind or minikube with the Docker driver, apply it on the host, because the nodes share the host kernel.

  2. Restart any process that already failed and fell back to polling. A restart makes it pick up inotify again:

sudo systemctl restart <service-name>

Tip: Each watch uses about 1 KB of kernel memory on 64-bit systems, so 524288 watches can use about 512 MB in the worst case. Size the limit to fit the node.

Verify#

Confirm the new values are active:

sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches

Stream logs again. The command should follow the logs without errors:

kubectl logs -f deployment/<name> -n <namespace>

tail should no longer print the polling warning:

tail -f /var/log/syslog