<div dir="ltr">Dear Mulyadi,<div><br></div><div>Another observation I wanted to add to this:</div><div><br></div><div>On some nodes, when the cached memory is high, for example when I look at the atop output, it showed me about 15G cached. Now when I run a simple malloc program to grab all available virtual memory and wait for OOM killer to kill it. To my surprise and obvious in retrospect I see all of my cached memory is now released and amount of free bumps up closer to the total RAM excluding the minimal used by the system services, which is what I would like to see. Now I thought If I use this simple trick it would clear up similar issues on other machines, but this was not true and the big difference was in the amount of cached memory. Another big difference between them are AnonHugePages and AnonPages. On the machine A below AnonHugePages is about 13.9G(13936640 kB), while on machine B below AnonHugePages is about 1.5G(1153024 kB). If I run my simple malloc program to infinitely claim virtual memory on the below two machines: Machine A's run will fail at 9G and no change on MemFree. And machine B's run will fail at 1.7TB (1735372 MB). Note the reclaimed space on Machine-B below after a simple malloc run to glob available virtual memory. </div><div><br></div><div>This basically tells me that AnonHugePages limit the amount of RAM available in the user space. Now I wonder what is using up the AnonPages on Machine-A and how can I claim that back without a reboot since there is no other user using the machine? Also is there a way I can peek into what process has allocated AnonPages so that I can tell if it is/was user related? Please advise.</div><div><br></div><div>-regards!!</div><div>Prem</div><div><br></div><div><br></div><div>A# free -g<br></div><div><div> total used free shared buffers cached</div><div>Mem: 23 14 8 0 0 0</div><div>-/+ buffers/cache: 14 9</div><div>Swap: 0 0 0</div></div><div><br></div><div><br></div><div>B# free -g</div><div> total used free shared buffers cached</div><div>Mem: 23 20 3 0 0 14</div><div>-/+ buffers/cache: 5 18</div><div>Swap: 0 0 0</div><div><br></div><div><br></div><div>MachineB after Malloc-run</div><div>$ free -g</div><div> total used free shared buffers cached</div><div>Mem: 23 2 21 0 0 0</div><div>-/+ buffers/cache: 1 21</div><div>Swap: 0 0 0</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 23, 2015 at 11:07 AM, Mulyadi Santosa <span dir="ltr"><<a href="mailto:mulyadi.santosa@gmail.com" target="_blank">mulyadi.santosa@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="h5"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 23, 2015 at 4:47 AM, Prem Kumar <span dir="ltr"><<a href="mailto:prem.it.kumar@gmail.com" target="_blank">prem.it.kumar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">also wondering if there is a way I can list Active memory map showing me what is cached?<div><br></div><div>-regards.</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 22, 2015 at 3:08 PM, Prem Kumar <span dir="ltr"><<a href="mailto:prem.it.kumar@gmail.com" target="_blank">prem.it.kumar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Dear All,<div><br></div><div>I have done quite a bit of reading on Active memory reported in /proc/meminfo and in short says it is never reclaimed unless absolutely necessary, and it caches the recently used files/pages in memory. Although I fail to understand the consequences that I face here. </div><div><br></div><div>I have disk-less and swap-less nodes. So all I have to do, is play with the RAM on the box. Issue that brought me here is investigating why after running some applications, used memory is never available for use with any other applications. </div><div><br></div><div>In other words I cannot run any programs that requests memory more than what is shown as free in the output of free command and MemFree in the output of the cat /proc/meminfo</div><div>For example if I ran any program that requires more than 6GB on the first node below and more than 1GB on the second node below they fail instantly, and work fine if within the limist of free. There is nothing else running on the system other than system processes/services. </div><div><br></div><div><div> total used free shared buffers cached</div><div>Mem: 23 17 6 0 0 9</div><div>-/+ buffers/cache: 8 15</div><div>Swap: 0 0 0</div><div><br></div><div> total used free shared buffers cached</div><div>Mem: 23 22 1 0 0 0</div><div>-/+ buffers/cache: 21 1</div><div>Swap: 0 0 0</div></div><div><br></div><div>Since the applications that ran previously are not running any more "even though they died out of memory because they requested more memory than available", shouldn't the OS see that any memory used previously as useless and can it not reclaim that for use with the next job/program on that machine. </div><div><br></div><div>On every machine that I have run into this problem the out put of /proc/meminfo shows that Active memory is used up the amount shown in the free command and limits my further runs. </div><div><br></div><div>This is driving me insane and making me feel stupid knowing that OS is smart enough to handle this, then what am I missing here to understand? Please advise. </div><div><br></div><div>Appreciate any insight into this. </div><div><br></div><div>Best Regards,</div><div>Prem</div><div><br></div><div><br></div></div>
</blockquote></div><br></div>
</div></div><br><br></blockquote></div><br></div></div></div><div class="gmail_extra">Dear Prem<br><br></div><div class="gmail_extra">welcome to kernelnewbies :) First of all, please don't do top posting when replying. Follow like what I and the rest of list member do.<br><br></div><div class="gmail_extra">Btw, looking from the free output, I have a doubt about your statement that your first application took 6 GB and secondly it took 1 GB. Assuming your application doesn't thing like memory locking in kernel space, i guess it takes 20+ GB of RAM.<br><br></div><div class="gmail_extra">So, before we go further, could you re run your applications and use ps or top to see both the VSIZE and RSS they take ?<br><br></div><div class="gmail_extra">Regarding memory claiming, yes after app is killed (using any ways possible: ctrl-c, sending kill/term/quit signal, OOM etc), any memory allocated by this task are freed. It happen on both active and inactive pages<span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><div class="gmail_extra"><br clear="all"><br>-- <br><div>regards,<br><br>Mulyadi Santosa<br>Freelance Linux trainer and consultant<br><br>blog: <a href="http://the-hydra.blogspot.com" target="_blank">the-hydra.blogspot.com</a><br>training: <a href="http://mulyaditraining.blogspot.com" target="_blank">mulyaditraining.blogspot.com</a></div>
</div></font></span></div>
</blockquote></div><br></div>