If ‘free’ is showing that a lot of memory is in use, even though all the processes in ‘ps’ don’t add up to all that much memory, then it might be the Linux kernel that’s using all of your memory, for caching. Ever since Linux kernel 2.2, the kernel avoids RAM fragmentation by managing it’s memory with an allocation technique called “slabs”. Take a look at the kernel’s “slab” memory:
# cat /proc/slabinfo
..or use ‘slabtop’.
The Linux kernel’s “VFS” virtual filesystem sub-system caches as much data about the filesystem in RAM as possible in order to speed access. It stores it in dynamically-allocated slabs. The inode and dentry caches speed up filesystem operations, by keeping the actual directory tree structure in RAM. Inodes are the objects that represent the underlying files (and also directories). A dentry is an object with a string name (d_name), a pointer to an inode (d_inode), and a pointer to the parent dentry (d_parent).
..you may see a huge “slab” for “dentry”:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 30333429 30333283 99% 0.19K 1444449 21 5777796K dentry
..voila – that’s where all the memory is going! So what IS a “dentry”?
Linux is supposed to give back that memory, when it sees that userspace processes need it, but…
“If you’ve got extremely aggressive processes that linux is killing because they’re trying to claim too much memory, tuning the cache cleanup parameters may help, but it’s unlikely. For that sort of information, see the vfs_cache_pressure documentation.”
So, what do the Linux kernel docs have to say about vfs_cache_pressure??
Controls the tendency of the kernel to reclaim the memory which is used for
caching of directory and inode objects.
At the default value of vfs_cache_pressure=100 the kernel will attempt to
reclaim dentries and inodes at a “fair” rate with respect to pagecache and
swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer
to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will
never reclaim dentries and inodes due to memory pressure and this can easily
lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100
causes the kernel to prefer to reclaim dentries and inodes.”
To check the current value of vfs_cache_pressure, try this:
$ sudo sysctl -n vm.vfs_cache_pressure 100 $
So, the bottom-line is: the kernel is supposed to free slabs as needed in order to make room for userspace applications, but if it encounters a process that’s doing a ton of filesystem operations, it might not free it’s VFS cache slabs fast enough, and if it runs out of memory, the kernel OOM killer will start killing processes, so if “free” (or New Relic) is showing that nearly all the physical memory is used up by slabs, you may need to change vfs_cache_pressure:
To change it:
$ sudo sysctl vm.vfs_cache_pressure=50 vm.vfs_cache_pressure = 150 $
To save the change, add a line to /etc/sysctl.conf such as:
vm.vfs_cache_pressure = 150