Missing Disk Space in Linux
Missing Disk Space in Linux
by Jeff on June 2nd, 2014 No Comments »
 , ,

I recently ran into a disk space issue on one of my CentOS 6 servers. The server was running out of space, however, this is a static server that rarely changes. I logged into the box and issued a df command

df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg_centos-lv_root   12G   12G  0M  100% /
tmpfs                          939M     0  939M   0% /dev/shm
/dev/sda1                      485M  117M  343M  26% /boot

I started down my usual troubleshooting path to see what was chewing up all of my disk space. I ran a series of du commands on the root (/) directory

du -sh /
2.4G    /

Hmm, if I’m only using 2.4gb, then why is df showing the disk is full? I ended up searching google for several hours and came across this
post on ServerFault. The post had me check for deleted files that may still have a handle open. The deleted files will not show up using the ‘du’ command, however, they will count against your disk space.

lsof +L1
COMMAND   PID    USER   FD   TYPE DEVICE   SIZE/OFF NLINK   NODE NAME
stunnel 32496 stunnel   10w   REG  253,0 8709152111     0 135331 /usr/local/var/lib/stunnel/stunnel.log (deleted)

And there’s my mysterious disk eating file. Apparently, a log file for Stunnel was still being written to and was now around 8.11gb. Add that to the 2.4 that du shows and some overhead, and you have just under 12gb.

Now the tricky part is removing that file. I have attempted to issue a

cat /dev/null > /usr/local/var/lib/stunnel/stunnel.log

Since the file does not exist, I cannot cat /dev/null into it to clear the data. I’ll have to restart the service and see if it clears out.


Leave a Reply

Your email address will not be published. Required fields are marked *