Hello, I have been tasked with writing a kernel module to handle an own protocol (with an own EtherType) in the Linux kernel. While I am not a complete newbie in the Linux kernel, having to handle Ethernet traffic there is new to me. I have looked at several tutorials and also at sample code (for example http://www.phrack.org/archives/55/p55_0x0c_Building%20Into%20The%20Linux%20N..., and some tutorials I cannot find at the moment...) and tried to cut it down to the bare essentials. I came up with the following code utilising Linux Protocol Module: http://www.trikaliotis.net/testmodule/test_kernel.c the correspondig Makefile is in http://www.trikaliotis.net/testmodule/Makefile Note: This is a cut-down version of the actual code which shows the problem. Additionally, I register for ETH_P_ALL so the problem occurs as fast as possible. While it essentially works as I want it to, I found that I am leaking memory this way. Having this module in a highly loaded network, "head -n5 /proc/meminfo" shows that MemFree is steadily falling, until the kernel OOPS with an out of memory condition. "Of course", if I undefine DO_MEMLEAK, the memory leak does not occur anymore. I tried for almost two weeks now to look at different tutorials, and I also looked in working code (for example, the IP LPM) and tried to compare with my code, but I could not find out what I am doing wrong here. I am sure it must be something obvious, and very silly - unfortunaely, for me, it is not obvious at the moment. Can anyone help me here, please? Best regards, Spiro. -- Spiro R. Trikaliotis http://opencbm.sf.net/ http://www.trikaliotis.net/ http://www.viceteam.org/
Hi! On 21:54 Fri 21 Jan , Spiro Trikaliotis wrote: ...
While it essentially works as I want it to, I found that I am leaking memory this way. Having this module in a highly loaded network, "head -n5 /proc/meminfo" shows that MemFree is steadily falling, until the kernel OOPS with an out of memory condition.
I could not see anything wrong here either. Some things which came to my mind where: - Does this also happen, if you do not call PromiscuousMode_Enter? - Which user grows is /proc/slabinfo? (If this file is emply or does not exist, you may meed to recompile the kernel to use slab instead of sl[b-z]b) - Which kernel version are you using? Have you tried different versions? -Michi -- programing a layer 3+4 network protocol for mesh networks see http://michaelblizek.twilightparadox.com
Hallo Michi, first of all, thank you for your answer. * On Sat, Jan 22, 2011 at 08:18:05AM +0100 Michael Blizek wrote:
On 21:54 Fri 21 Jan , Spiro Trikaliotis wrote:
- Does this also happen, if you do not call PromiscuousMode_Enter?
Yes, it does. In fact, I only included it in my minimal example so the memory leak happens faster, in case someone wants to give it a try himself.
- Which user grows is /proc/slabinfo? (If this file is emply or does not exist, you may meed to recompile the kernel to use slab instead of sl[b-z]b)
Thank you for the pointer, it might help. The objects which have changed the most on a mildly loaded network are: 25000 buffer_head 5000 dentry 5000 ext3_inode_cache 5000 size-64 The number if the number of more active object after approx. 20h of letting it run on the mildly loaded network. Note that the ext3_inode_cache might have grown because I was regularly writing a new log file from slabinfo (slabinfo --once > slabinfo.`date +...`) Thus, I would expect I am leaking buffer_head.
- Which kernel version are you using? Have you tried different versions?
I am running this on a Debian Lenny (32 bit), Kernel 2.6.26-2-686, and an Ubuntu 10.04, also 32 bit. Unfortunately this machine is currently out of my reach, but I think it is a variant of 2.6.32 or 2.6.33. Both kernels are from the distribution, that is, no vanilla kernels. Beste Grüße Spiro. -- Spiro R. Trikaliotis http://opencbm.sf.net/ http://www.trikaliotis.net/ http://www.viceteam.org/
Hi! On 11:40 Sun 23 Jan , Spiro Trikaliotis wrote: ...
* On Sat, Jan 22, 2011 at 08:18:05AM +0100 Michael Blizek wrote: ...
- Which user grows is /proc/slabinfo? (If this file is emply or does not exist, you may meed to recompile the kernel to use slab instead of sl[b-z]b)
Thank you for the pointer, it might help.
The objects which have changed the most on a mildly loaded network are:
25000 buffer_head 5000 dentry 5000 ext3_inode_cache 5000 size-64
The number if the number of more active object after approx. 20h of letting it run on the mildly loaded network.
Note that the ext3_inode_cache might have grown because I was regularly writing a new log file from slabinfo (slabinfo --once > slabinfo.`date +...`)
Thus, I would expect I am leaking buffer_head.
The funny point is that buffer_head belongs to the filesystem subsystem. What file systems are you using? Could it be that your code just triggers the memory leak, because data is logged to disk, e.g. to /var/log/kern.log ? I have tried your program on my virtual machine (2.6.28) and could see any leak, but maybe data is leaked very slowly... -Michi -- programing a layer 3+4 network protocol for mesh networks see http://michaelblizek.twilightparadox.com
Hi!
On 11:40 Sun 23 Jan , Spiro Trikaliotis wrote: ...
* On Sat, Jan 22, 2011 at 08:18:05AM +0100 Michael Blizek wrote: ...
- Which user grows is /proc/slabinfo? (If this file is emply or does not exist, you may meed to recompile the kernel to use slab instead of sl[b-z]b)
Thank you for the pointer, it might help.
The objects which have changed the most on a mildly loaded network are:
25000 buffer_head 5000 dentry 5000 ext3_inode_cache 5000 size-64
The number if the number of more active object after approx. 20h of letting it run on the mildly loaded network.
Note that the ext3_inode_cache might have grown because I was regularly writing a new log file from slabinfo (slabinfo --once > slabinfo.`date +...`)
Thus, I would expect I am leaking buffer_head.
The funny point is that buffer_head belongs to the filesystem subsystem. What file systems are you using? Could it be that your code just triggers the memory leak, because data is logged to disk, e.g. to /var/log/kern.log ? I did small experiment with your code.I removed all your logs which were getting logged in kernel buffers(kern.log).With this change i checked the "meminfo" and found that the memory leaking is almost same as compared to normal case(with no change in your code).
I have tried your program on my virtual machine (2.6.28) and could see any leak, but maybe data is leaked very slowly...
i confirm that data is leaking very slowly (below is the output i got on my ubuntu machine with removed logs from your code). $date Sun Jan 23 19:13:47 RET 2011 $ head -n5 /proc/meminfo MemTotal: 1018172 kB MemFree: 373708 kB Buffers: 32232 kB Cached: 296004 kB SwapCached: 0 kB $ date Sun Jan 23 19:15:03 RET 2011 $ head -n5 /proc/meminfo MemTotal: 1018172 kB MemFree: 373584 kB Buffers: 32248 kB Cached: 296004 kB SwapCached: 0 kB With logs enabled in your code i can see marginal increase in leaking memory. I can defintely see the memory leaking with your ko and will investigate further as to the reason. Hopefully kmemleak will lead us somewhere.
-Michi -- programing a layer 3+4 network protocol for mesh networks see http://michaelblizek.twilightparadox.com
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi! On 21:22 Sun 23 Jan , anish kumar wrote:
The funny point is that buffer_head belongs to the filesystem subsystem. What file systems are you using? Could it be that your code just triggers the memory leak, because data is logged to disk, e.g. to /var/log/kern.log ? I did small experiment with your code.I removed all your logs which were getting logged in kernel buffers(kern.log).With this change i checked the "meminfo" and found that the memory leaking is almost same as compared to normal case(with no change in your code).
I have tried your program on my virtual machine (2.6.28) and could see any leak, but maybe data is leaked very slowly...
i confirm that data is leaking very slowly (below is the output i got on my ubuntu machine with removed logs from your code).
$date Sun Jan 23 19:13:47 RET 2011
$ head -n5 /proc/meminfo MemTotal: 1018172 kB MemFree: 373708 kB Buffers: 32232 kB Cached: 296004 kB SwapCached: 0 kB
$ date Sun Jan 23 19:15:03 RET 2011
$ head -n5 /proc/meminfo MemTotal: 1018172 kB MemFree: 373584 kB Buffers: 32248 kB Cached: 296004 kB SwapCached: 0 kB
With logs enabled in your code i can see marginal increase in leaking memory.
This small increase could easily be caused by background activity. Please try to reproduce this in single user mode with all file systems mounted read-only (use e.g. "mount -o remount,ro /") and no processes running except init and your shell. Then see if memory increase is both correlated to network traffic and reason of system crashes. If it takes too long, you can add e.g. "mem=32M" to the kernel boot parameter list in the bootloader. -Michi -- programing a layer 3+4 network protocol for mesh networks see http://michaelblizek.twilightparadox.com
Hi!
On 21:22 Sun 23 Jan , anish kumar wrote:
The funny point is that buffer_head belongs to the filesystem subsystem. What file systems are you using? Could it be that your code just triggers the memory leak, because data is logged to disk, e.g. to /var/log/kern.log ? I did small experiment with your code.I removed all your logs which were getting logged in kernel buffers(kern.log).With this change i checked the "meminfo" and found that the memory leaking is almost same as compared to normal case(with no change in your code).
I have tried your program on my virtual machine (2.6.28) and could see any leak, but maybe data is leaked very slowly...
i confirm that data is leaking very slowly (below is the output i got on my ubuntu machine with removed logs from your code).
$date Sun Jan 23 19:13:47 RET 2011
$ head -n5 /proc/meminfo MemTotal: 1018172 kB MemFree: 373708 kB Buffers: 32232 kB Cached: 296004 kB SwapCached: 0 kB
$ date Sun Jan 23 19:15:03 RET 2011
$ head -n5 /proc/meminfo MemTotal: 1018172 kB MemFree: 373584 kB Buffers: 32248 kB Cached: 296004 kB SwapCached: 0 kB
With logs enabled in your code i can see marginal increase in leaking memory.
This small increase could easily be caused by background activity. Please try to reproduce this in single user mode with all file systems mounted read-only (use e.g. "mount -o remount,ro /") and no processes running except init and your shell. Then see if memory increase is both correlated to network traffic and reason of system crashes. If it takes too long, you can add e.g. "mem=32M" to the kernel boot parameter list in the bootloader. Thanks for this suggestions.
After persuading my kernel to use only 100M(with 32M system not booting,it is not vanilla kernel) of my physical memory and after installing your ko i tried to check meminfo and found that the claim made by you(.ko is leaking memory) may not be true as i can see in the output of /proc/meminfo,MemFree is sometimes increasing and sometimes decreasing(could be because of several other components). I can say with certainty that your ko looks fine to me after this testing. I couldn't see any panic happening in my system( system is running fine under such low memory). However i see that several other applications(xserv) is crashing because of out of memory which i think is fine. ps:I dont have vanilla kernel where i could have changed the intitialisation files to remount while booting and could have controlled the list of process running on the system. In android we are doing that in init.rc file need to check in ubuntu.
-Michi -- programing a layer 3+4 network protocol for mesh networks see http://michaelblizek.twilightparadox.com
Hi! On 22:59 Sun 23 Jan , anish kumar wrote: ...
ps:I dont have vanilla kernel where i could have changed the intitialisation files to remount while booting and could have controlled the list of process running on the system. In android we are doing that in init.rc file need to check in ubuntu.
Why not type "init 1" in a root shell? -Michi -- programing a layer 3+4 network protocol for mesh networks see http://michaelblizek.twilightparadox.com
On Mon, Jan 24, 2011 at 9:38 PM, Michael Blizek < michi1@michaelblizek.twilightparadox.com> wrote:
Hi!
On 22:59 Sun 23 Jan , anish kumar wrote: ...
ps:I dont have vanilla kernel where i could have changed the intitialisation files to remount while booting and could have controlled the list of process running on the system. In android we are doing that in init.rc file need to check in ubuntu.
Why not type "init 1" in a root shell?
Thanks for this information but looks like this is not properly supported in my ubuntu system as after this command it is dropping to recovery menu.
-Michi -- programing a layer 3+4 network protocol for mesh networks see http://michaelblizek.twilightparadox.com
Hi Spiro,
While it essentially works as I want it to, I found that I am leaking memory this way. Having this module in a highly loaded network, "head -n5 /proc/meminfo" shows that MemFree is steadily falling, until the kernel OOPS with an out of memory condition.
Have you considered using kmemleak [1] to help your debugging? thanks, Daniel. [1] http://lxr.linux.no/linux+v2.6.37/Documentation/kmemleak.txt
participants (5)
-
anish kumar -
anish singh
-
Daniel Baluta -
Michael Blizek -
Spiro Trikaliotis