Ubuntu – General way to monitor ALL memory usages

freememory usagezfs

I have Kubuntu 16.04 and am using ZFS.

As you can see on superuser, I just had a long journey in tracking down where my memory was vanishing to.

It turned out to be the ARC of ZFS that was eating it up without showing up anywhere in the tools/files other than its own special file /proc/spl/kstat/zfs/arcstats

Is there a generic way to see how much memory is used by the kernel and if possible, by which kernel modules?

Example:

free -h was showing me this:

              total        used        free      shared  buff/cache   available
Mem:            31G         19G        1.8G        406M         10G        5.2G
Swap:          8.0G          0B        8.0G


So 19GB in use. But TOP only showed me about 8GB of used memory. So I went in search of the other 11GB.
6GB turned out to be slab which can be seen in /proc/memstat and with nmon. But 5GB were still missing.

In the end I found that it had been consumed by the ARC of ZFS. But it was only listed in its own special info file. /proc/spl/kstat/zfs/arcstats

Neither /proc/memstat nor top, htop, nmon, free, or atop could tell me who is using those 5GB.

Is there a file/tool to see the memory used by such special modules, without knowing/understanding those modules? Or do I have to account for every existing module if I would want to write a monitoring tool that can tell me this?

From my perspective, memory info under linux is a total mess. Dozen of number that all don't add up to total memory. I bet free calculates the "in use" parameter by substracting free/cache/buffers from the total.

I would like to have/write a tool/info where I can see

Total memory: 32GB
in use: 19GB
processes: 8GB
slab: 5GB
mapper: 1Gb
kernel modules: 5Gb
kernel: 0.3GB
spl: 4.7GB
cache: 6GB
buffers: 2GB
free: 5GB
Active memory: 8.3GB
Active(anon): 7.1GB
Active(file): 1.2GB
Inactive memory: 2.4GB
...


You know, where each subsection actually addes up to its parent section and each byte of memory is accounted for 😉

So is such a tool possible under Linux? Does it exist already?

Update 1

The tool smem as suggested below indeed shows a colum "Dynamic kernel memory" which sums up to 100% of memory and accounts for ARC.

BTW: I already checked smem sources, it basically does (memtotal – userspace – free – cache).

So this is only achieved by simply substracting all known figures from /proc/memstat and labeling what is missing as "noncache Dynamic Kernel Memory" I find this disturbing and really hope the linux kernel has more to offer than that.

• You might try smem for at least a goodly portion of what you are looking for.

It's available in the Universe repository and can be installed with sudo apt-get install smem

There are a number of useful switches that are covered on the man page man smem

Some examples are as follows: Note I'm using the -t switch for totals in all examples for brevity. you can leave it out if you want more verbose output.

PSS is the interesting column in these examples as it takes shared memory into account.


System Wide

me@zippy-64bit:~$smem -tw Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 989656 860076 129580 userspace memory 936032 180020 756012 free memory 5692716 5692716 0 ---------------------------------------------------------- 7618404 6732812 885592  By user me@zippy-64bit:~$ smem -tu
User     Count     Swap      USS      PSS      RSS
me          67        0   861452   898915  1245592
---------------------------------------------------
67        0   861452   898915  1245592


Also of note are smemcap for capture of data for future analysis with smem and smemstat which reports the physical memory usage taking into consideration shared memory. The tool can either report a current snapshot of memory usage or periodically dump out any changes in memory.

There's also a list at the bottom of the page of similar packages.

Sources:

Alternative 1: You might also consider parsing the output of cat /proc/meminfo or if you want it by the page less /proc/meminfo

Much of the information here is used by free, top and ps commands. In fact, the output of the free command is similar in appearance to the contents and structure of /proc/meminfo. But by looking directly at /proc/meminfo, many more details are revealed including :

\$ cat /proc/meminfo | grep "Slab"
Slab:             267680 kB


Alternative 2: If you are truly committed to a better parsing of kernel RAM you might roll your own kstat. While kstat appears to be a Solaris only command. the link above will take you to sources for for an open source attempt at kstat that was last modified in 2013 (as of this edit.) A clever C programmer might be able to modify the code for their specific purpose.

Alternative 2 source: