Issue#

Writes fail with "no space left", but df -h shows plenty of free space on the filesystem:

touch: cannot touch '/var/www/app/cache/test': No space left on device
$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda1 50G 18G 30G 38% /

Services may also fail to start, apt may fail to unpack packages, and logs may stop being written.

Cause#

Every file, directory and symlink uses one inode. On ext4, the number of inodes is fixed when the filesystem is created. If something creates millions of small files, the inodes run out before the blocks do. Common causes are PHP session files, cache directories, mail queues (/var/spool/postfix), and old build artifacts. Once all inodes are used, the kernel returns ENOSPC even though blocks are still free.

Solution#

  1. Confirm that inodes are exhausted:
df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 3276800 3276800 0 100% /
  1. Find the directories that hold the most inodes. -x keeps du on one filesystem:
sudo du --inodes -x / 2>/dev/null | sort -n | tail -n 15
  1. Drill into the worst directory to find the culprit:
sudo du --inodes -x /var/lib/php 2>/dev/null | sort -n | tail -n 10 sudo find /var/lib/php/sessions -xdev -type f | wc -l
  1. Delete the unneeded files. Do not use rm *, because it fails with "Argument list too long". Use find -delete instead. This example removes session files older than one day:
sudo find /var/lib/php/sessions -xdev -type f -mmin +1440 -delete

For a stuck Postfix queue, flush or purge it with Postfix's own tools:

sudo postqueue -p | tail -n 1 sudo postsuper -d ALL deferred
  1. Fix the source so the problem does not return. For example, add a tmpfiles rule that cleans a cache directory automatically:
echo 'e /var/www/app/cache - - - 7d' | sudo tee /etc/tmpfiles.d/app-cache.conf sudo systemd-tmpfiles --clean
  1. If the workload really does need many small files, give it more inodes. On ext4, you cannot add inodes to an existing filesystem. You have to recreate it with a smaller bytes-per-inode ratio (-i) or an explicit inode count (-N). This erases the filesystem, so back up the data first:
sudo mkfs.ext4 -i 8192 /dev/sdb1

On XFS, inodes are allocated dynamically. You can raise the maximum inode percentage online:

sudo xfs_growfs -m 50 /mountpoint

Verify#

Inode usage should be well below 100%, and writes should succeed again:

df -i / touch /var/www/app/cache/test && rm /var/www/app/cache/test && echo OK

Tip: Monitor IUse% alongside Use%. Most disk alerts only watch block usage and miss inode exhaustion until something breaks.