Question on Memory Leak: very tough problem

[Bernd Petrovitsch]

Hi!

On Mit, 2011-10-26 at 23:20 -0400, Sri Ram Vemulpali wrote:
> I have very difficult problem I am working on to figure out where the
> memory leak is in my system. Sometimes it shows up and sometimes it
> reveals very slowly (I mean increments very slowly). This is dependent
> on system load, if the system is in very high load, then leak is very
> fast. We tried solving this problem from many dimensions, but no
> success.
> 
> After quite sometime, we started using valgrind to capture the memory
> leak. The funny thing is the leak disappears once we start running our
> application on top of valgrind.

You do speak of an user-space application?

> So, we cannot capture leak using valgrind. We came to conclusion that,
> since valgrind serializes the allocations (leak disappears), we
> thought this might be result of race conditions, that
> resulted from system high load (by the way ours is multi threaded
> system). We run system without valgrind we see the memory leak.

That really smells like a race condition. So you should make sure that
you got the synchronization between the threads correct and also the
pointer handling in your application. The memory leak seems to be either
a forgotten free() or an overwritten already allocated pointer as in
----  snip ----
void whatever(....)
{
	static void *p = NULL;
	if (p == NULL) {
		p = malloc(12345);
	}
}
----  snip ----
If that function is called in 2 threads in parallel and a task switch
occurs after the check for NULL and before the assignment afterwards,
you've got a memory leak.

That pattern works (at least) also for global pointers and pointers in
shared data structures.

> The question is, is it true that valgrind serializes the mem
> allocations. Does this happen to anyone disappearing the leak when
> used valgrind.

The manual page of malloc() doesn't mentions anything on threads so I
assume that malloc() is synchronized with itself (read: it is
thread-safe) and free(),realloc(),calloc(),strdup(), and all similar
functions.

Apart from valgrind, there are other memory debugging libs out there
(e.g. dmalloc) which may have far less impact on the systems
performance.
Depending on the skill, time and desperation level, building an
(somewhat old) gcc with an applied "bounds-checking" patch may help more
(but you may also run in Heisenberg-problems ....).

	Bernd
-- 
Bernd Petrovitsch                  Email : bernd at petrovitsch.priv.at
                     LUGA : http://www.luga.at