Active Memory never reclaimed please help me understand
Dear All, I have done quite a bit of reading on Active memory reported in /proc/meminfo and in short says it is never reclaimed unless absolutely necessary, and it caches the recently used files/pages in memory. Although I fail to understand the consequences that I face here. I have disk-less and swap-less nodes. So all I have to do, is play with the RAM on the box. Issue that brought me here is investigating why after running some applications, used memory is never available for use with any other applications. In other words I cannot run any programs that requests memory more than what is shown as free in the output of free command and MemFree in the output of the cat /proc/meminfo For example if I ran any program that requires more than 6GB on the first node below and more than 1GB on the second node below they fail instantly, and work fine if within the limist of free. There is nothing else running on the system other than system processes/services. total used free shared buffers cached Mem: 23 17 6 0 0 9 -/+ buffers/cache: 8 15 Swap: 0 0 0 total used free shared buffers cached Mem: 23 22 1 0 0 0 -/+ buffers/cache: 21 1 Swap: 0 0 0 Since the applications that ran previously are not running any more "even though they died out of memory because they requested more memory than available", shouldn't the OS see that any memory used previously as useless and can it not reclaim that for use with the next job/program on that machine. On every machine that I have run into this problem the out put of /proc/meminfo shows that Active memory is used up the amount shown in the free command and limits my further runs. This is driving me insane and making me feel stupid knowing that OS is smart enough to handle this, then what am I missing here to understand? Please advise. Appreciate any insight into this. Best Regards, Prem
also wondering if there is a way I can list Active memory map showing me what is cached? -regards. On Tue, Sep 22, 2015 at 3:08 PM, Prem Kumar <prem.it.kumar@gmail.com> wrote:
Dear All,
I have done quite a bit of reading on Active memory reported in /proc/meminfo and in short says it is never reclaimed unless absolutely necessary, and it caches the recently used files/pages in memory. Although I fail to understand the consequences that I face here.
I have disk-less and swap-less nodes. So all I have to do, is play with the RAM on the box. Issue that brought me here is investigating why after running some applications, used memory is never available for use with any other applications.
In other words I cannot run any programs that requests memory more than what is shown as free in the output of free command and MemFree in the output of the cat /proc/meminfo For example if I ran any program that requires more than 6GB on the first node below and more than 1GB on the second node below they fail instantly, and work fine if within the limist of free. There is nothing else running on the system other than system processes/services.
total used free shared buffers cached Mem: 23 17 6 0 0 9 -/+ buffers/cache: 8 15 Swap: 0 0 0
total used free shared buffers cached Mem: 23 22 1 0 0 0 -/+ buffers/cache: 21 1 Swap: 0 0 0
Since the applications that ran previously are not running any more "even though they died out of memory because they requested more memory than available", shouldn't the OS see that any memory used previously as useless and can it not reclaim that for use with the next job/program on that machine.
On every machine that I have run into this problem the out put of /proc/meminfo shows that Active memory is used up the amount shown in the free command and limits my further runs.
This is driving me insane and making me feel stupid knowing that OS is smart enough to handle this, then what am I missing here to understand? Please advise.
Appreciate any insight into this.
Best Regards, Prem
On Wed, Sep 23, 2015 at 4:47 AM, Prem Kumar <prem.it.kumar@gmail.com> wrote:
also wondering if there is a way I can list Active memory map showing me what is cached?
-regards.
On Tue, Sep 22, 2015 at 3:08 PM, Prem Kumar <prem.it.kumar@gmail.com> wrote:
Dear All,
I have done quite a bit of reading on Active memory reported in /proc/meminfo and in short says it is never reclaimed unless absolutely necessary, and it caches the recently used files/pages in memory. Although I fail to understand the consequences that I face here.
I have disk-less and swap-less nodes. So all I have to do, is play with the RAM on the box. Issue that brought me here is investigating why after running some applications, used memory is never available for use with any other applications.
In other words I cannot run any programs that requests memory more than what is shown as free in the output of free command and MemFree in the output of the cat /proc/meminfo For example if I ran any program that requires more than 6GB on the first node below and more than 1GB on the second node below they fail instantly, and work fine if within the limist of free. There is nothing else running on the system other than system processes/services.
total used free shared buffers cached Mem: 23 17 6 0 0 9 -/+ buffers/cache: 8 15 Swap: 0 0 0
total used free shared buffers cached Mem: 23 22 1 0 0 0 -/+ buffers/cache: 21 1 Swap: 0 0 0
Since the applications that ran previously are not running any more "even though they died out of memory because they requested more memory than available", shouldn't the OS see that any memory used previously as useless and can it not reclaim that for use with the next job/program on that machine.
On every machine that I have run into this problem the out put of /proc/meminfo shows that Active memory is used up the amount shown in the free command and limits my further runs.
This is driving me insane and making me feel stupid knowing that OS is smart enough to handle this, then what am I missing here to understand? Please advise.
Appreciate any insight into this.
Best Regards, Prem
Dear Prem welcome to kernelnewbies :) First of all, please don't do top posting when replying. Follow like what I and the rest of list member do. Btw, looking from the free output, I have a doubt about your statement that your first application took 6 GB and secondly it took 1 GB. Assuming your application doesn't thing like memory locking in kernel space, i guess it takes 20+ GB of RAM. So, before we go further, could you re run your applications and use ps or top to see both the VSIZE and RSS they take ? Regarding memory claiming, yes after app is killed (using any ways possible: ctrl-c, sending kill/term/quit signal, OOM etc), any memory allocated by this task are freed. It happen on both active and inactive pages -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Dear Mulyadi, Thank you for your response. Sorry for top posting it, I was waiting for my posting to arrive in my mail-box but it took for ever and hence top posted in eagerness. You are right in your observation that I couldn't possibly have had my first application crash at 6GB. I should have said about 15GB. I have many nodes I just picked the outputs from couple and presented by observation. Below here I will try to dissect my observation in the hope you can help me understand this, my OS concepts have become a little and haven't been in touch with them. Here is a machine that Currently has this state: $ free -g total used free shared buffers cached Mem: 23 14 8 0 0 0 -/+ buffers/cache: 14 9 Swap: 0 0 0 I have a program that just globs memory here is what happens when I run this: $ ./eatmemory 8.99G Eating 8589934592 bytes in chunks of 1024... Done, press any key to free the memory $ ./eatmemory 9G Eating 9663676416 bytes in chunks of 1024... Killed I believe the above observation is nothing wrong, because RAM is used by what other(assuming running) applications and I only have so much available for my program to run. But my issue is nothing else other than system services are running on this machine, this renders this node un-usable for the next program that runs on this machine and when request more than what 9G as above. Below here is the output of /proc/meminfo from the same machine $ cat /proc/meminfo MemTotal: 24724728 kB MemFree: 9402768 kB Buffers: 0 kB Cached: 217464 kB SwapCached: 0 kB Active: 14650896 kB Inactive: 60456 kB Active(anon): 14647052 kB Inactive(anon): 40632 kB Active(file): 3844 kB Inactive(file): 19824 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 14493928 kB Mapped: 19544 kB Shmem: 193720 kB Slab: 109720 kB SReclaimable: 12300 kB SUnreclaim: 97420 kB KernelStack: 2968 kB PageTables: 39100 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 12362364 kB Committed_AS: 15684044 kB VmallocTotal: 34359738367 kB VmallocUsed: 493316 kB VmallocChunk: 34346062668 kB HardwareCorrupted: 0 kB AnonHugePages: 13936640 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 7652 kB DirectMap2M: 25145344 kB Also here is my ulimit which is unlimited: $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 192912 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 81920 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited And /proc/self/maps $ cat /proc/self/maps 00400000-0040b000 r-xp 00000000 00:10 67455633 /bin/cat 0060a000-0060b000 rw-p 0000a000 00:10 67455633 /bin/cat 0060b000-0060c000 rw-p 00000000 00:00 0 0080a000-0080b000 rw-p 0000a000 00:10 67455633 /bin/cat 0209f000-020c0000 rw-p 00000000 00:00 0 [heap] 36d7e00000-36d7e20000 r-xp 00000000 00:10 67454760 /lib64/ld-2.12.so 36d801f000-36d8020000 r--p 0001f000 00:10 67454760 /lib64/ld-2.12.so 36d8020000-36d8021000 rw-p 00020000 00:10 67454760 /lib64/ld-2.12.so 36d8021000-36d8022000 rw-p 00000000 00:00 0 36d8200000-36d838a000 r-xp 00000000 00:10 67456999 /lib64/libc-2.12.so 36d838a000-36d858a000 ---p 0018a000 00:10 67456999 /lib64/libc-2.12.so 36d858a000-36d858e000 r--p 0018a000 00:10 67456999 /lib64/libc-2.12.so 36d858e000-36d858f000 rw-p 0018e000 00:10 67456999 /lib64/libc-2.12.so 36d858f000-36d8594000 rw-p 00000000 00:00 0 7f754caad000-7f754cab0000 rw-p 00000000 00:00 0 7f754cac2000-7f754cac3000 rw-p 00000000 00:00 0 7fff5e496000-7fff5e4ab000 rw-p 00000000 00:00 0 [stack] 7fff5e5f8000-7fff5e5f9000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] On every machine i ran into this problem, anonpages are eating up the memory,in effect shrinking the available RAM for the programs to run. Q) Now my question is since the previous job/program that ran on this machines has finished or died: My OS concepts tell me that the recently used cached-anonpages will be released to meet the request of another application requesting to use up the memory/vm. What am I missing here to understand? Also what I fail to understand is the state in which my diskelss & swapless nodes remain: What/who has control over the used up memory, why is it not being granted for the next owner of the machine to run at full scale? I understand that I will not have all of it but at least 19GB out of 24GB. Also below is the list of top process on the machines: Looking at it I don't see any heave use of memory ...mystery make me feel dumb?? $ ps aux --sort -rss USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 8402 0.0 0.0 119712 15896 ? S 12:11 0:00 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --debug-to-files root 9555 0.0 0.0 3508796 4680 ? S Aug18 0:33 /usr/sbin/slurmd useralap 8231 0.0 0.0 27224 4604 pts/0 S 12:06 0:00 -bash root 8401 0.0 0.0 151052 4264 ? S 12:11 0:00 /usr/libexec/sssd/sssd_be --domain default --uid 0 --gid 0 --debug-to-files root 8153 0.0 0.0 111192 3240 pts/0 Ss 12:05 0:00 -bash root 2078 0.0 0.0 720600 2968 ? Ssl Aug10 1:39 automount --pid-file /var/run/autofs.pid root 1752 0.0 0.0 249344 2784 ? Sl Aug10 0:04 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5 useralap 9898 1.0 0.0 26196 1468 pts/0 R+ 15:42 0:00 ps aux --sort -rss root 8150 0.0 0.0 111816 1296 ? Ss 12:05 0:00 sshd: root@pts/0 munge 2146 0.0 0.0 225004 1292 ? Sl Aug10 0:36 /usr/sbin/munged 68 2006 0.0 0.0 41976 1228 ? Ssl Aug10 0:10 hald root 1671 0.0 0.0 9120 976 ? Ss Aug10 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient-em1.leases -pf /var/run/dhclient-em1.pid root 8400 0.0 0.0 114288 900 ? Ss 12:11 0:00 /usr/sbin/sssd -f -D root 8403 0.0 0.0 105264 876 ? S 12:11 0:00 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --debug-to-files root 2174 0.0 0.0 20000 868 ? Ss Aug10 0:42 crond root 2111 0.0 0.0 66188 712 ? Ss Aug10 0:00 /usr/sbin/sshd root 3625 0.0 0.0 334616 648 ? SLsl Aug10 0:00 /usr/sbin/ibacm root 863 0.0 0.0 10832 592 ? S<s Aug10 0:00 /sbin/udevd -d root 3391 0.0 0.0 10828 588 ? S< Aug10 0:00 /sbin/udevd -d root 3523 0.0 0.0 10828 588 ? S< Aug10 0:00 /sbin/udevd -d rpcuser 1893 0.0 0.0 25428 464 ? Ss Aug10 0:00 rpc.statd root 1736 0.0 0.0 93176 460 ? S<sl Aug10 0:07 auditd root 1 0.0 0.0 23500 452 ? Ss Aug10 0:02 /sbin/init root 1799 0.0 0.0 10912 452 ? Ss Aug10 6:45 irqbalance --pid=/var/run/irqbalance.pid root 8230 0.0 0.0 165156 448 pts/0 S 12:06 0:00 su - useralap rpc 1875 0.0 0.0 18976 300 ? Ss Aug10 0:02 rpcbind dbus 1934 0.0 0.0 23484 280 ? Ss Aug10 0:00 dbus-daemon --system root 2199 0.0 0.0 21076 212 ? Ss Aug10 0:00 /usr/sbin/atd root 2207 0.0 0.0 21792 212 ? S Aug10 0:24 /usr/sbin/ipmievd sel pidfile=/var/run/ipmievd.pid root 2007 0.0 0.0 20400 184 ? S Aug10 0:00 hald-runner root 2043 0.0 0.0 22520 164 ? S Aug10 0:00 hald-addon-input: Listening on /dev/input/event0 68 2045 0.0 0.0 18008 148 ? S Aug10 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket root 1997 0.0 0.0 4080 116 ? Ss Aug10 0:00 /usr/sbin/acpid root 2097 0.0 0.0 6260 116 ? Ss Aug10 0:00 /usr/sbin/mcelog --daemon root 2222 0.0 0.0 4064 76 tty2 Ss+ Aug10 0:00 /sbin/mingetty /dev/tty2 Please advise and let me know if you need more information. -best regards!! On Wed, Sep 23, 2015 at 11:07 AM, Mulyadi Santosa <mulyadi.santosa@gmail.com
wrote:
On Wed, Sep 23, 2015 at 4:47 AM, Prem Kumar <prem.it.kumar@gmail.com> wrote:
also wondering if there is a way I can list Active memory map showing me what is cached?
-regards.
On Tue, Sep 22, 2015 at 3:08 PM, Prem Kumar <prem.it.kumar@gmail.com> wrote:
Dear All,
I have done quite a bit of reading on Active memory reported in /proc/meminfo and in short says it is never reclaimed unless absolutely necessary, and it caches the recently used files/pages in memory. Although I fail to understand the consequences that I face here.
I have disk-less and swap-less nodes. So all I have to do, is play with the RAM on the box. Issue that brought me here is investigating why after running some applications, used memory is never available for use with any other applications.
In other words I cannot run any programs that requests memory more than what is shown as free in the output of free command and MemFree in the output of the cat /proc/meminfo For example if I ran any program that requires more than 6GB on the first node below and more than 1GB on the second node below they fail instantly, and work fine if within the limist of free. There is nothing else running on the system other than system processes/services.
total used free shared buffers cached Mem: 23 17 6 0 0 9 -/+ buffers/cache: 8 15 Swap: 0 0 0
total used free shared buffers cached Mem: 23 22 1 0 0 0 -/+ buffers/cache: 21 1 Swap: 0 0 0
Since the applications that ran previously are not running any more "even though they died out of memory because they requested more memory than available", shouldn't the OS see that any memory used previously as useless and can it not reclaim that for use with the next job/program on that machine.
On every machine that I have run into this problem the out put of /proc/meminfo shows that Active memory is used up the amount shown in the free command and limits my further runs.
This is driving me insane and making me feel stupid knowing that OS is smart enough to handle this, then what am I missing here to understand? Please advise.
Appreciate any insight into this.
Best Regards, Prem
Dear Prem
welcome to kernelnewbies :) First of all, please don't do top posting when replying. Follow like what I and the rest of list member do.
Btw, looking from the free output, I have a doubt about your statement that your first application took 6 GB and secondly it took 1 GB. Assuming your application doesn't thing like memory locking in kernel space, i guess it takes 20+ GB of RAM.
So, before we go further, could you re run your applications and use ps or top to see both the VSIZE and RSS they take ?
Regarding memory claiming, yes after app is killed (using any ways possible: ctrl-c, sending kill/term/quit signal, OOM etc), any memory allocated by this task are freed. It happen on both active and inactive pages
-- regards,
Mulyadi Santosa Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Dear Mulyadi, Another observation I wanted to add to this: On some nodes, when the cached memory is high, for example when I look at the atop output, it showed me about 15G cached. Now when I run a simple malloc program to grab all available virtual memory and wait for OOM killer to kill it. To my surprise and obvious in retrospect I see all of my cached memory is now released and amount of free bumps up closer to the total RAM excluding the minimal used by the system services, which is what I would like to see. Now I thought If I use this simple trick it would clear up similar issues on other machines, but this was not true and the big difference was in the amount of cached memory. Another big difference between them are AnonHugePages and AnonPages. On the machine A below AnonHugePages is about 13.9G(13936640 kB), while on machine B below AnonHugePages is about 1.5G(1153024 kB). If I run my simple malloc program to infinitely claim virtual memory on the below two machines: Machine A's run will fail at 9G and no change on MemFree. And machine B's run will fail at 1.7TB (1735372 MB). Note the reclaimed space on Machine-B below after a simple malloc run to glob available virtual memory. This basically tells me that AnonHugePages limit the amount of RAM available in the user space. Now I wonder what is using up the AnonPages on Machine-A and how can I claim that back without a reboot since there is no other user using the machine? Also is there a way I can peek into what process has allocated AnonPages so that I can tell if it is/was user related? Please advise. -regards!! Prem A# free -g total used free shared buffers cached Mem: 23 14 8 0 0 0 -/+ buffers/cache: 14 9 Swap: 0 0 0 B# free -g total used free shared buffers cached Mem: 23 20 3 0 0 14 -/+ buffers/cache: 5 18 Swap: 0 0 0 MachineB after Malloc-run $ free -g total used free shared buffers cached Mem: 23 2 21 0 0 0 -/+ buffers/cache: 1 21 Swap: 0 0 0 On Wed, Sep 23, 2015 at 11:07 AM, Mulyadi Santosa <mulyadi.santosa@gmail.com
wrote:
On Wed, Sep 23, 2015 at 4:47 AM, Prem Kumar <prem.it.kumar@gmail.com> wrote:
also wondering if there is a way I can list Active memory map showing me what is cached?
-regards.
On Tue, Sep 22, 2015 at 3:08 PM, Prem Kumar <prem.it.kumar@gmail.com> wrote:
Dear All,
I have done quite a bit of reading on Active memory reported in /proc/meminfo and in short says it is never reclaimed unless absolutely necessary, and it caches the recently used files/pages in memory. Although I fail to understand the consequences that I face here.
I have disk-less and swap-less nodes. So all I have to do, is play with the RAM on the box. Issue that brought me here is investigating why after running some applications, used memory is never available for use with any other applications.
In other words I cannot run any programs that requests memory more than what is shown as free in the output of free command and MemFree in the output of the cat /proc/meminfo For example if I ran any program that requires more than 6GB on the first node below and more than 1GB on the second node below they fail instantly, and work fine if within the limist of free. There is nothing else running on the system other than system processes/services.
total used free shared buffers cached Mem: 23 17 6 0 0 9 -/+ buffers/cache: 8 15 Swap: 0 0 0
total used free shared buffers cached Mem: 23 22 1 0 0 0 -/+ buffers/cache: 21 1 Swap: 0 0 0
Since the applications that ran previously are not running any more "even though they died out of memory because they requested more memory than available", shouldn't the OS see that any memory used previously as useless and can it not reclaim that for use with the next job/program on that machine.
On every machine that I have run into this problem the out put of /proc/meminfo shows that Active memory is used up the amount shown in the free command and limits my further runs.
This is driving me insane and making me feel stupid knowing that OS is smart enough to handle this, then what am I missing here to understand? Please advise.
Appreciate any insight into this.
Best Regards, Prem
Dear Prem
welcome to kernelnewbies :) First of all, please don't do top posting when replying. Follow like what I and the rest of list member do.
Btw, looking from the free output, I have a doubt about your statement that your first application took 6 GB and secondly it took 1 GB. Assuming your application doesn't thing like memory locking in kernel space, i guess it takes 20+ GB of RAM.
So, before we go further, could you re run your applications and use ps or top to see both the VSIZE and RSS they take ?
Regarding memory claiming, yes after app is killed (using any ways possible: ctrl-c, sending kill/term/quit signal, OOM etc), any memory allocated by this task are freed. It happen on both active and inactive pages
-- regards,
Mulyadi Santosa Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
participants (2)
-
Mulyadi Santosa -
Prem Kumar