Hi! On Mit, 2011-10-26 at 23:20 -0400, Sri Ram Vemulpali wrote:

I have very difficult problem I am working on to figure out where the memory leak is in my system. Sometimes it shows up and sometimes it reveals very slowly (I mean increments very slowly). This is dependent on system load, if the system is in very high load, then leak is very fast. We tried solving this problem from many dimensions, but no success.

After quite sometime, we started using valgrind to capture the memory leak. The funny thing is the leak disappears once we start running our application on top of valgrind.

You do speak of an user-space application?

So, we cannot capture leak using valgrind. We came to conclusion that, since valgrind serializes the allocations (leak disappears), we thought this might be result of race conditions, that resulted from system high load (by the way ours is multi threaded system). We run system without valgrind we see the memory leak.

That really smells like a race condition. So you should make sure that you got the synchronization between the threads correct and also the pointer handling in your application. The memory leak seems to be either a forgotten free() or an overwritten already allocated pointer as in ---- snip ---- void whatever(....) { static void *p = NULL; if (p == NULL) { p = malloc(12345); } } ---- snip ---- If that function is called in 2 threads in parallel and a task switch occurs after the check for NULL and before the assignment afterwards, you've got a memory leak. That pattern works (at least) also for global pointers and pointers in shared data structures.

The question is, is it true that valgrind serializes the mem allocations. Does this happen to anyone disappearing the leak when used valgrind.

The manual page of malloc() doesn't mentions anything on threads so I assume that malloc() is synchronized with itself (read: it is thread-safe) and free(),realloc(),calloc(),strdup(), and all similar functions. Apart from valgrind, there are other memory debugging libs out there (e.g. dmalloc) which may have far less impact on the systems performance. Depending on the skill, time and desperation level, building an (somewhat old) gcc with an applied "bounds-checking" patch may help more (but you may also run in Heisenberg-problems ....). Bernd -- Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at