Disk full but du shows free space: find deleted files still held open

Issue#

A server reports a full disk and applications fail to write, even after you deleted large log files:

write error: No space left on device

df says the filesystem is full, but du reports much less usage:

$ df -h /var
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        50G   50G     0 100% /var

$ sudo du -sh /var
12G     /var

Cause#

On Linux, deleting a file (rm) only removes its directory entry. If a process still has the file open, the kernel keeps the data blocks allocated until that process closes the file descriptor or exits.

This usually happens when a large log file is deleted with rm while the service writing to it, such as nginx, java, or rsyslog, keeps running. du walks the directory tree, so it cannot see the file anymore. df asks the filesystem, which still counts the blocks.

Solution#

List open files that have been deleted (link count less than 1):

sudo lsof +L1

Example output:

COMMAND   PID USER   FD   TYPE DEVICE    SIZE/OFF NLINK   NODE NAME
java     1834  app    7w   REG    8,3 38654705664     0 131090 /var/log/app/app.log (deleted)

The SIZE/OFF column shows the space being held (about 36G here). The FD column shows the file descriptor number (7).

If lsof is not installed, find the same files through /proc:

sudo find /proc/*/fd -ls 2>/dev/null | grep '(deleted)'

Preferred fix: restart or reload the service so it closes the old file and opens a new one:

sudo systemctl restart app.service

For services that reopen logs on a signal, a reload is enough:

sudo systemctl reload nginx

If you cannot restart the process, truncate the deleted file through its file descriptor. Use the PID and FD number from step 1:

sudo truncate -s 0 /proc/1834/fd/7

This frees the blocks immediately and leaves the process running.

To prevent this from happening again, empty active log files instead of deleting them:

sudo truncate -s 0 /var/log/app/app.log

You can also configure logrotate with copytruncate for applications that cannot reopen their logs:

/var/log/app/*.log {
    daily
    rotate 7
    compress
    copytruncate
}

Verify#

Confirm that no large deleted files are still held open:

sudo lsof +L1

Check that df and du now roughly agree:

df -h /var
sudo du -sh /var

Tip: If df -h shows free space but you still get No space left on device, check inode usage with df -i. Millions of small files can exhaust inodes before the disk runs out of blocks.