Dear Mulyadi,
Another observation I wanted to add to this:
On some nodes, when the cached memory is high, for example when I look at the atop output, it showed me about 15G cached. Now when I run a simple malloc program to grab all available virtual memory and wait for OOM killer to kill it. To my surprise and obvious in retrospect I see all of my cached memory is now released and amount of free bumps up closer to the total RAM excluding the minimal used by the system services, which is what I would like to see. Now I thought If I use this simple trick it would clear up similar issues on other machines, but this was not true and the big difference was in the amount of cached memory. Another big difference between them are AnonHugePages and AnonPages. On the machine A below AnonHugePages is about 13.9G(13936640 kB), while on machine B below AnonHugePages is about 1.5G(1153024 kB). If I run my simple malloc program to infinitely claim virtual memory on the below two machines: Machine A's run will fail at 9G and no change on MemFree. And machine B's run will fail at 1.7TB (1735372 MB). Note the reclaimed space on Machine-B below after a simple malloc run to glob available virtual memory.
This basically tells me that AnonHugePages limit the amount of RAM available in the user space. Now I wonder what is using up the AnonPages on Machine-A and how can I claim that back without a reboot since there is no other user using the machine? Also is there a way I can peek into what process has allocated AnonPages so that I can tell if it is/was user related? Please advise.
-regards!!
Prem
A# free -g
total used free shared buffers cached
Mem: 23 14 8 0 0 0
-/+ buffers/cache: 14 9
Swap: 0 0 0
B# free -g
total used free shared buffers cached
Mem: 23 20 3 0 0 14
-/+ buffers/cache: 5 18
Swap: 0 0 0
MachineB after Malloc-run
$ free -g
total used free shared buffers cached
Mem: 23 2 21 0 0 0
-/+ buffers/cache: 1 21
Swap: 0 0 0