I recently ran into a disk space issue on one of my CentOS 6 servers. The server was running out of space, however, this is a static server that rarely changes. I logged into the box and issued a df command
df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_centos-lv_root 12G 12G 0M 100% / tmpfs 939M 0 939M 0% /dev/shm /dev/sda1 485M 117M 343M 26% /boot
I started down my usual troubleshooting path to see what was chewing up all of my disk space. I ran a series of du commands on the root (/) directory
du -sh / 2.4G /
Hmm, if I’m only using 2.4gb, then why is df showing the disk is full? I ended up searching google for several hours and came across this
post on ServerFault. The post had me check for deleted files that may still have a handle open. The deleted files will not show up using the ‘du’ command, however, they will count against your disk space.
lsof +L1 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME stunnel 32496 stunnel 10w REG 253,0 8709152111 0 135331 /usr/local/var/lib/stunnel/stunnel.log (deleted)
And there’s my mysterious disk eating file. Apparently, a log file for Stunnel was still being written to and was now around 8.11gb. Add that to the 2.4 that du shows and some overhead, and you have just under 12gb.
Now the tricky part is removing that file. I have attempted to issue a
cat /dev/null > /usr/local/var/lib/stunnel/stunnel.log
Since the file does not exist, I cannot cat /dev/null into it to clear the data. I’ll have to restart the service and see if it clears out.