Disk issues – No space left on device

Reading time: 7m

If your disk is full, and your website goes down, don’t panic!  There’s always a solution!

The Problem

My website works, but I can't login/logout!
I can't send any emails!
No space left on device

If you’ve ever complained about that, it’s likely one of two things has happened, as described below.

Disk is full

It’s easy to forget how much disk space it takes to run a full server with all the bells and whistles.  Unless you use software like cPanel, you’d never receive any notification from the server that your disk is almost full.  When you login to your server and run df -h, you see:

Filesystem                 Size  Used Avail Use% Mounted on
/dev/disk/by-label/ROOT     40G   40G    0M 100% /
udev                        10M     0   10M   0% /dev
tmpfs                      403M   41M  362M  11% /run
tmpfs                     1007M     0 1007M   0% /dev/shm
tmpfs                      5.0M     0  5.0M   0% /run/lock
tmpfs                     1007M     0 1007M   0% /sys/fs/cgroup

Where did all the space go?  Lucky for us it’s easy to find out!

If you’re running Debian, Ubuntu, or CentOS, then you can install a package called ncdu which is an ncurses version of the du command (which calculates file and folder size)

For Debian and Ubuntu:

sudo apt-get install ncdu

For CentOS:

yum install ncdu

Once it’s installed, then run the following command to get an overview of what files are taking up space on the / partition (or actually any partition by name):

ncdu -x /

This will show you a graph, sorted by largest size, with the folder name in the rightmost column as shown below.  The -x option tells ncdu not to calculate files or folders that cross filesystem boundaries (e.g. files on /var if it were on it’s own partition).  I didn’t actually have a full server, but this screenshot should show you what it looks like:

ncdu output

Pressing Shift+C will add a column between the size and the percentage bar, indicating the number of child items.  Screenshot below:

ncdu output with inodesYou can navigate through the directories using Left to go up a directory, Right or Enter to enter a directory, Up and Down to scroll through the list, and the “d” key to Delete a folder.

Inode Table is full

If you’re having the same issue, but when you go to your server to df -h, it shows plenty of space available, then you have a slightly different problem.  Chances are you’ve filled the inode table so it can no longer add any references.  An inode is a reference to a file, folder, symbolic link, character device, or socket file.

For this you have to use a different df flag, namely:

df -i

This will show you a little different output than the command from the previous section:

Filesystem                Inodes   IUsed   IFree IUse% Mounted on
/dev/disk/by-label/ROOT  2621440 2621440      0  100% /
udev                      256156     269 255887    1% /dev
tmpfs                     257649     386 257263    1% /run
tmpfs                     257649       1 257648    1% /dev/shm
tmpfs                     257649       3 257646    1% /run/lock
tmpfs                     257649      11 257638    1% /sys/fs/cgroup

In this case, we’d run the same command as before, but this time make use of Shift+C to turn on child counts for directories.  This will also sort the list in descending numeric order for you, so the directories with the most inode counts float to the top.  Drill down a few directories until you see few numbers that are high and most that are low, and you’ll have found the culprit.  What to do with those files is up to you at that point!

Leave a Reply

Your email address will not be published. Required fields are marked *