Root partition is full


Introduction

The system’s root partition (also known as / or slash) is the partition where the operating system’s base is installed.

Every other user files should be located in a sub-directory of /home.

When the root partition is near or reach its maximum capacity it can translate into an array of hard to diagnose problems on your server.

Symptoms

  • The system is unstable
  • Apache cannot be restarted
  • Email server cannot fetch emails
  • Impossible to install new software or updates

Troubleshooting

With a simple SSH command:

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 1.9G 981M 887M 53% /
/dev/hda2 34G 3.8G 29G 12% /home
none 61M 0 61M 0% /dev/shm

Or if you are using Webmin:

  • With SSL: https://nsXXXXXXX.ip-XXX-XXX-XXX.net:10000/fdisk/
  • Without SSL: http://nsXXXXXXX.ip-XXX-XXX-XXX.net:10000/fdisk/

Note

You must replace nsXXXXXXX.ip-XXX-XXX-XXX.net with the hostname of your server.

Causes

They are many:

  • Backup of your sites in a directory other than a /home sub-directory;
  • Big archives extracted in the /root directory;
  • Setup of massive software (for example online game server) in a directory belonging to the system partition (for example /usr),
  • Bug of mod_gzip;
  • Overflow of the email queue
  • Old logs piled up

Solutions

Unloading

For the first two cases, you just need to move the files/directories in a /home sub-directory.

First we need to find large files that need to be moved elsewhere. For this you can use the command du.

Here’s some examples of how du can be used:

ls
a.log  b.log  c.log  d.log
du
3411980 .
du -chs
3.3G    .
3.3G    total
du -chs *
2.0M    a.log
30M     b.log
301M    c.log
3.0G    d.log
3.3G    total

To find the five most massive files in a directory, you can execute the following command:

ls -lh --sort=size | head -n 5
total 20M
-rw-r--r-- 1 root root 2.6M may 1 18:00 vim-common-6.1-18.7x.2.i386.rpm
-rw-r--r-- 1 root root 2.5M jui 22 10:15 mutt-1.4.1-1.src.rpm
-rw-r--r-- 1 root root 2.5M jui 22 10:33 mutt-1.4.1i-2mdk.src.rpm
-rw-r--r-- 1 root root 1.7M apr 9 2003 bind-9.2.1-1.7x.2.i386.rpm
-rw-r--r-- 1 root root 1.5M jui 18 14:25 tin-current.tar.gz

Here, we note that:

There is a 20MB file in this directory (the sub directories are not taken into account) The more massive file is vim-common-6.1-18.7x.2.i386.rpm: its size is 2.6MB

To solve the problem, we are going to place the bulky files in a directory on /home.

Visibly, these are .rpm and .tar.gz (softwares archives). You are not obliged to keep them in /root. Therefore, we are going to creae a directory on /home to store them.

mkdir /home/archives
mv *.tar.gz *.rpm /home/archives

Unloading massive software

For example, you have installed a HLDS server (Half-Life Dedicated Server) in /usr/local/game/hlds. This directory is on the system partition, thus you have to move it on /home and create a symbolic link in order the path to remain valid.

Warning

Make sure you stop the HLDS server before the operation!

mv /usr/local/game/hlds /home/
ln -s /home/hlds /usr/local/game/hlds

Bug of mod_gzip

It may happen that mod_gzip doesn’t delete its temporary files located in /tmp and that they reach an important size (several GB). To realise it:

ls -l /tmp/*.wrk
-rwx------ 1 nobody nobody 53695415 sep 30 00:10 _11831_132_33.wrk
-rwx------ 1 nobody nobody 0 sep 3 00:10 _12954_120_21.wrk
-rwx------ 1 nobody nobody 0 sep 7 00:10 _14733_110_11.wrk
-rwx------ 1 nobody nobody 0 sep 21 00:10 _16191_106_7.wrk
-rwx------ 1 nobody nobody 0 aoû 19 00:10 _16585_123_24.wrk
-rwx------ 1 nobody nobody 0 aoû 25 00:10 _16693_152_53.wrk
-rwx------ 1 nobody nobody 0 oct 5 00:10 _17282_110_11.wrk
-rwx------ 1 nobody nobody 0 sep 14 00:10 _17792_106_7.wrk
-rwx------ 1 nobody nobody 0 oct 7 00:10 _18056_108_9.wrk

To delete them:

rm -rf /tmp/*.wrk

This bug is known but its origin is not. For the moment, the only one solution to avoid it is to deactivate mod_gzip in httpd.conf. Look for mod_gzip yes line and replace it by mod_gzip no. Then restart Apache that way:

/etc/init.d/httpd restart

This bug only occurs for some site (probably due to a PHP or CGI script).