Heap memory is not re-claiming.
Hello all, I've a daemon process and I'm allocating heap memory** for big character buffers using malloc/free. Each can take 5MB. Though I freed/deleted memory allocated for the buffers, the increased memory during the allocation time is not re-claiming back.This I observed using *top* command. I'm sure I do not have any other memory leaks. I debugged the code line by line and observing memory usage using *top* command. During allocation time, *top* command showed that 30 MB increase in Virtual & Resident memory. Upon execution of delete/free statement on the buffers the raised memory was not re-claimed. On subsequent execution of the same code no extra memory was allocated even though new/malloc statements called on the buffers.What is kernel mechanism here. Is this due to some page cache re-mapping If this due to page cache then the memory should always constant trough out process life. During long run the memory slowly increasing. This should not happen. Could some body provide any insights into this.. Regards, Ravi
On Thu, Oct 13, 2011 at 1:26 AM, V.Ravikumar <ravikumar.vallabhu@gmail.com> wrote:
I've a daemon process and I'm allocating heap memory for big character buffers using malloc/free. Each can take 5MB.
Though I freed/deleted memory allocated for the buffers, the increased memory during the allocation time is not re-claiming back.This I observed using top command.
Yes, I think it's glibc - it may keep your memory, with the idea that you'll request it again soon anyway. I don't know if there's a way to force it to give the memory back to Linux.
On Thu, Oct 13, 2011 at 2:12 PM, Jeff Donner <jeffrey.donner@gmail.com>wrote:
On Thu, Oct 13, 2011 at 1:26 AM, V.Ravikumar <ravikumar.vallabhu@gmail.com> wrote:
I've a daemon process and I'm allocating heap memory for big character buffers using malloc/free. Each can take 5MB.
Though I freed/deleted memory allocated for the buffers, the increased memory during the allocation time is not re-claiming back.This I observed using top command.
Yes, I think it's glibc - it may keep your memory, with the idea that you'll request it again soon anyway.
If this is the case then memory should not keep on increase albeit how long process may run. right?
On Thu, Oct 13, 2011 at 1:26 AM, V.Ravikumar <ravikumar.vallabhu@gmail.com> wrote:
I've a daemon process and I'm allocating heap memory for big character buffers using malloc/free. Each can take 5MB.
Though I freed/deleted memory allocated for the buffers, the increased memory during the allocation time is not re-claiming back.This I observed using top command.
Yes, I think it's glibc - it may keep your memory, with the idea that you'll request it again soon anyway.
If this is the case then memory should not keep on increase albeit how long process may run. right?
Right. That's a memory leak, or else you're just not freeing what you think you are. It's a user-land program, right? 'valgrind' is awesome for finding leaks and memory still allocated. If it really isn't your program losing memory someone else will have to jump in but, I can't imagine any reason a non-broken kernel would accumulate resources.
On Thu, Oct 13, 2011 at 2:36 PM, V.Ravikumar <ravikumar.vallabhu@gmail.com>wrote:
On Thu, Oct 13, 2011 at 2:12 PM, Jeff Donner <jeffrey.donner@gmail.com>wrote:
On Thu, Oct 13, 2011 at 1:26 AM, V.Ravikumar <ravikumar.vallabhu@gmail.com> wrote:
I've a daemon process and I'm allocating heap memory for big character buffers using malloc/free. Each can take 5MB.
Though I freed/deleted memory allocated for the buffers, the increased memory during the allocation time is not re-claiming back.This I observed using top command.
Yes, I think it's glibc - it may keep your memory, with the idea that you'll request it again soon anyway.
If this is the case then memory should not keep on increase albeit how long process may run. right?
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi Ravi,
When you ask for a memory block, usually by using malloc(), you're asking the runtime C library whether a preallocated block is available. This block's size must *at least* equal the user request. If there is already a memory block available, malloc() will assign this block to the user and mark it as "used." Otherwise, malloc() must allocate more memory by extending the heap. All requested blocks go in an area called the *heap. Reference to an article by Mulayadi Santosa :- http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html AWESOME ARTICLE SIR :) *Regards, Rohan Puri
Nice doc ...:) On Fri, Oct 14, 2011 at 10:28 AM, rohan puri <rohan.puri15@gmail.com> wrote:
On Thu, Oct 13, 2011 at 2:36 PM, V.Ravikumar <ravikumar.vallabhu@gmail.com> wrote:
On Thu, Oct 13, 2011 at 2:12 PM, Jeff Donner <jeffrey.donner@gmail.com> wrote:
On Thu, Oct 13, 2011 at 1:26 AM, V.Ravikumar <ravikumar.vallabhu@gmail.com> wrote:
I've a daemon process and I'm allocating heap memory for big character buffers using malloc/free. Each can take 5MB.
Though I freed/deleted memory allocated for the buffers, the increased memory during the allocation time is not re-claiming back.This I observed using top command.
Yes, I think it's glibc - it may keep your memory, with the idea that you'll request it again soon anyway.
If this is the case then memory should not keep on increase albeit how long process may run. right?
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi Ravi,
When you ask for a memory block, usually by using malloc(), you're asking the runtime C library whether a preallocated block is available. This block's size must at least equal the user request. If there is already a memory block available, malloc() will assign this block to the user and mark it as "used." Otherwise, malloc() must allocate more memory by extending the heap. All requested blocks go in an area called the heap.
Reference to an article by Mulayadi Santosa :-
http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
AWESOME ARTICLE SIR :)
Regards, Rohan Puri
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-- Pankaj SIngh Phone No: 9921865080
Hi.... :) On Fri, Oct 14, 2011 at 16:15, pankaj singh <psingh.ait@gmail.com> wrote:
Nice doc ...:)
On Fri, Oct 14, 2011 at 10:28 AM, rohan puri <rohan.puri15@gmail.com> wrote:
Reference to an article by Mulayadi Santosa :-
http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
AWESOME ARTICLE SIR :)
Thank you, thank you very much for your appreciation :) I just hope, besides getting adequate money, you all got something meaningful from that article. And I believe, as long as there are people who have same concerned, that article will be refined over and over and overall will be better through the time :) -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
On Sat, Oct 15, 2011 at 9:31 AM, Mulyadi Santosa <mulyadi.santosa@gmail.com>wrote:
Hi.... :)
On Fri, Oct 14, 2011 at 16:15, pankaj singh <psingh.ait@gmail.com> wrote:
Nice doc ...:)
On Fri, Oct 14, 2011 at 10:28 AM, rohan puri <rohan.puri15@gmail.com> wrote:
Reference to an article by Mulayadi Santosa :-
http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
AWESOME ARTICLE SIR :)
Thank you, thank you very much for your appreciation :) I just hope, besides getting adequate money, you all got something meaningful from that article. And I believe, as long as there are people who have same concerned, that article will be refined over and over and overall will be better through the time :)
-- regards,
Mulyadi Santosa Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi, It's really a fabulous article. Many many thanks to Mulyadi for such a great article and thank for Rohan for sharing the link. I read the article and correlate contents with my current issue. Here are my observations. Do please correct me if I'm wrong at any point. The leak which I was observed in my program is due to big memory buffers of size 5MB allocation from heap.I've 6 such character buffers. So as per below lines from the article memory should be allocated with mmap and it immediately releases memory to kernel upon free/delete call from user land process. The allocator uses two functions to get a chunk of memory from the kernel: - brk() sets the end of the process's data segment. - mmap() creates a new VMA and passes it to the allocator. The decision on whether to use brk() or mmap() requires one simple check. If the request is equal or larger than M_MMAP_THRESHOLD, the allocator uses mmap(). If it is smaller, the allocator calls brk(). By default, M_MMAP_THRESHOLD is 128KB But It seems I'm landed with a case which uses brk() for allocation and so it just marks as free when I my program frees the memory and hence 30 MB memory (I've 6 , 5MB buffers from heap)was kept on allocator's control leading to a 30MB leak of user land process. Please note that I'm freeing memory for 6 buffers (I'm sure about that). Regards, Ravi
On Tue, Oct 18, 2011 at 10:04 AM, V.Ravikumar <ravikumar.vallabhu@gmail.com>wrote:
On Sat, Oct 15, 2011 at 9:31 AM, Mulyadi Santosa < mulyadi.santosa@gmail.com> wrote:
Hi.... :)
On Fri, Oct 14, 2011 at 16:15, pankaj singh <psingh.ait@gmail.com> wrote:
Nice doc ...:)
On Fri, Oct 14, 2011 at 10:28 AM, rohan puri <rohan.puri15@gmail.com> wrote:
Reference to an article by Mulayadi Santosa :-
http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
AWESOME ARTICLE SIR :)
Thank you, thank you very much for your appreciation :) I just hope, besides getting adequate money, you all got something meaningful from that article. And I believe, as long as there are people who have same concerned, that article will be refined over and over and overall will be better through the time :)
-- regards,
Mulyadi Santosa Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi,
It's really a fabulous article. Many many thanks to Mulyadi for such a great article and thank for Rohan for sharing the link.
I read the article and correlate contents with my current issue. Here are my observations. Do please correct me if I'm wrong at any point.
The leak which I was observed in my program is due to big memory buffers of size 5MB allocation from heap.I've 6 such character buffers.
So as per below lines from the article memory should be allocated with mmap and it immediately releases memory to kernel upon free/delete call from user land process.
The allocator uses two functions to get a chunk of memory from the kernel:
- brk() sets the end of the process's data segment. - mmap() creates a new VMA and passes it to the allocator.
The decision on whether to use brk() or mmap() requires one simple check. If the request is equal or larger than M_MMAP_THRESHOLD, the allocator uses mmap(). If it is smaller, the allocator calls brk(). By default, M_MMAP_THRESHOLD is 128KB
But It seems I'm landed with a case which uses brk() for allocation and so it just marks as free when I my program frees the memory and hence 30 MB memory (I've 6 , 5MB buffers from heap)was kept on allocator's control leading to a 30MB leak of user land process. Please note that I'm freeing memory for 6 buffers (I'm sure about that).
Regards, Ravi
Yes I think thats the case.
Regards, Rohan Puri
On Sat, Oct 15, 2011 at 9:31 AM, Mulyadi Santosa <mulyadi.santosa@gmail.com>wrote:
Hi.... :)
On Fri, Oct 14, 2011 at 16:15, pankaj singh <psingh.ait@gmail.com> wrote:
Nice doc ...:)
On Fri, Oct 14, 2011 at 10:28 AM, rohan puri <rohan.puri15@gmail.com> wrote:
Reference to an article by Mulayadi Santosa :-
http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
AWESOME ARTICLE SIR :)
Thank you, thank you very much for your appreciation :) I just hope, besides getting adequate money, you all got something meaningful from that article. And I believe, as long as there are people who have same concerned, that article will be refined over and over and overall will be better through the time :)
-- regards,
Mulyadi Santosa Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Right :) Regards, Rohan Puri
On Fri, Oct 14, 2011 at 10:28 AM, rohan puri <rohan.puri15@gmail.com> wrote:
*Reference to an article by Mulayadi Santosa :-
http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html *
Rohan, thanks a lot for sharing this link. Mulyadi, the article is just superb, very well-written and extremely useful :) Regards, Srivatsa S. Bhat
participants (6)
-
Jeff Donner -
Mulyadi Santosa -
pankaj singh -
rohan puri -
Srivatsa Bhat -
V.Ravikumar