If your disk is full, and your website goes down, don’t panic! There’s always a solution!
The Problem
My website works, but I can't login/logout!
I can't send any emails!
No space left on device
If you’ve ever complained about that, it’s likely one of two things has happened, as described below.
Disk is full
It’s easy to forget how much disk space it takes to run a full server with all the bells and whistles. Unless you use software like cPanel, you’d never receive any notification from the server that your disk is almost full. When you login to your server and run df -h, you see:
Filesystem Size Used Avail Use% Mounted on /dev/disk/by-label/ROOT 40G 40G 0M 100% / udev 10M 0 10M 0% /dev tmpfs 403M 41M 362M 11% /run tmpfs 1007M 0 1007M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 1007M 0 1007M 0% /sys/fs/cgroup
Where did all the space go? Lucky for us it’s easy to find out!
If you’re running Debian, Ubuntu, or CentOS, then you can install a package called ncdu which is an ncurses version of the du command (which calculates file and folder size)
For Debian and Ubuntu:
sudo apt-get install ncdu
For CentOS:
yum install ncdu
Once it’s installed, then run the following command to get an overview of what files are taking up space on the / partition (or actually any partition by name):
ncdu -x /
This will show you a graph, sorted by largest size, with the folder name in the rightmost column as shown below. The -x option tells ncdu not to calculate files or folders that cross filesystem boundaries (e.g. files on /var if it were on it’s own partition). I didn’t actually have a full server, but this screenshot should show you what it looks like:
Pressing Shift+C will add a column between the size and the percentage bar, indicating the number of child items. Screenshot below:
You can navigate through the directories using Left to go up a directory, Right or Enter to enter a directory, Up and Down to scroll through the list, and the “d” key to Delete a folder.
Inode Table is full
If you’re having the same issue, but when you go to your server to df -h, it shows plenty of space available, then you have a slightly different problem. Chances are you’ve filled the inode table so it can no longer add any references. An inode is a reference to a file, folder, symbolic link, character device, or socket file.
For this you have to use a different df flag, namely:
df -i
This will show you a little different output than the command from the previous section:
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/disk/by-label/ROOT 2621440 2621440 0 100% / udev 256156 269 255887 1% /dev tmpfs 257649 386 257263 1% /run tmpfs 257649 1 257648 1% /dev/shm tmpfs 257649 3 257646 1% /run/lock tmpfs 257649 11 257638 1% /sys/fs/cgroup
In this case, we’d run the same command as before, but this time make use of Shift+C to turn on child counts for directories. This will also sort the list in descending numeric order for you, so the directories with the most inode counts float to the top. Drill down a few directories until you see few numbers that are high and most that are low, and you’ll have found the culprit. What to do with those files is up to you at that point!