Debugging techniques inside kernel
I am not completely aware of the debugging techniques and utilizing /proc entries that kernel provides to debug the issues. Could you please point me out to some interesting sources related to debugging or it would be very nice if you could list them here at one place. I found out that recently some interesting CONFIGS that enable some debugging features. For ex : *CONFIG_SLUB_DEBUG_ON, *CONFIG_DMA_API_DEBUG I am just told to use these configs and check. Not sure how to use them effectively. Please share if you are aware of these kind of configs and how to use them. To have them at one place it would be useful for debugging beginners like me. Currently i am facing a memory corruption issue. A very Nasty one which has different call stacks everytime it crashes. Looking at the patterns and Area that is getting corrupted, we are suspecting non-linux components that are illegally writing into the comple DDR(ram). So in this case, is there some particular help that kernel can provide me? Currently i have the complete ram dump. I am analyzing it through opensource crash tool. Cheers, Manty!
On Mon, 08 Jul 2013 11:22:16 +0900, manty kuma said:
For ex : *CONFIG_SLUB_DEBUG_ON, *CONFIG_DMA_API_DEBUG I am just told to use these configs and check. Not sure how to use them effectively. Please share if you are aware of these kind of configs and how to use them. To have them at one place it would be useful for debugging beginners like me.
Most _DEBUG config options are set-and-forget - you turn them on and there's no further configuration needed. They enable additional code that does additional sanity checking and if it finds a problem, it issues a printk() or a stack dump or similar which shows up in dmesg. If you're chasing memory corruption, it may be useful to use netconsole or a serial console to send out all the printk() output in real time, as often an error will be detected but the system will panic()/crash/hang in a way that your local syslog daemon is unable to write the message to disk. (I'm told that on newer UEFI based systems, you can use pstore in a similar fashion, but have not tried it myself yet).
Hi Valdis, Thanks for the info. I have serial logs (using minicom). This is not helping me much as randomly memory is getting crashed and everytime it changes. Even the logs before the crash are varying everytime. On Mon, Jul 8, 2013 at 11:49 AM, <Valdis.Kletnieks@vt.edu> wrote:
On Mon, 08 Jul 2013 11:22:16 +0900, manty kuma said:
For ex : *CONFIG_SLUB_DEBUG_ON, *CONFIG_DMA_API_DEBUG I am just told to use these configs and check. Not sure how to use them effectively. Please share if you are aware of these kind of configs and how to use them. To have them at one place it would be useful for debugging beginners like me.
Most _DEBUG config options are set-and-forget - you turn them on and there's no further configuration needed. They enable additional code that does additional sanity checking and if it finds a problem, it issues a printk() or a stack dump or similar which shows up in dmesg.
If you're chasing memory corruption, it may be useful to use netconsole or a serial console to send out all the printk() output in real time, as often an error will be detected but the system will panic()/crash/hang in a way that your local syslog daemon is unable to write the message to disk. (I'm told that on newer UEFI based systems, you can use pstore in a similar fashion, but have not tried it myself yet).
On 7/8/13, manty kuma <mantykuma@gmail.com> wrote:
I am not completely aware of the debugging techniques and utilizing /proc entries that kernel provides to debug the issues.
Could you please point me out to some interesting sources related to debugging or it would be very nice if you could list them here at one place.
I found out that recently some interesting CONFIGS that enable some debugging features.
For ex : *CONFIG_SLUB_DEBUG_ON, *CONFIG_DMA_API_DEBUG I am just told to use these configs and check. Not sure how to use them effectively. Please share if you are aware of these kind of configs and how to use them. To have them at one place it would be useful for debugging beginners like me.
Currently i am facing a memory corruption issue. A very Nasty one which has different call stacks everytime it crashes. Looking at the patterns and Area that is getting corrupted, we are suspecting non-linux components that are illegally writing into the comple DDR(ram). So in this case, is there some particular help that kernel can provide me?
I've had a similar issue and the reason was a corrupted memory chip.
Currently i have the complete ram dump. I am analyzing it through opensource crash tool.
Cheers, Manty!
-- Regards, Denis
From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies-bounces@kernelnewbies.org] On Behalf Of manty kuma Sent: Monday, July 08, 2013 7:52 AM To: kernelnewbies@kernelnewbies.org Subject: Debugging techniques inside kernel
I am not completely aware of the debugging techniques and utilizing /proc entries that kernel provides to debug the issues. Could you please point me out to some interesting sources related to debugging or it would be very nice if you could list them here at one place.
I found out that recently some interesting CONFIGS that enable some debugging features. For ex : CONFIG_SLUB_DEBUG_ON, CONFIG_DMA_API_DEBUG I am just told to use these configs and check. Not sure how to use them effectively. Please share if you are aware of these kind of configs and how to use them. To have them at one place it would be useful for debugging beginners like me.
Currently i am facing a memory corruption issue. A very Nasty one which has different call stacks everytime it crashes. Looking at the patterns and Area that is getting corrupted, we are suspecting non-linux components that are illegally writing into the comple DDR(ram). So in this case, is there some particular help that kernel can provide me? Currently i have the complete ram dump. I am analyzing it through opensource crash tool.
I am no expert in debugging the memory corruptions. Also I have no knowledge about the Config options you mentioned. One thing you could try out is - *if possible*, eliminate the non-standard Kernel modules one by one so you can isolate the module causing the issue. Also try out Kfence (http://www.fsl.cs.stonybrook.edu/docs/kefence-storagess05/kefence.html) and Config option DEBUG_STACKOVERFLOW - this will help identify the exact place when the buffer is overrun (and not when the corrupted memory is accessed). Enabling the edac driver also might help, in case of any hardware issues. And finally the kgdb (http://www.stlinux.com/devel/debug/kgdb/debugging-the-kernel) - I haven't tried it myself. Hope this helps.
Cheers, Manty!
participants (4)
-
Denis Kirjanov -
manty kuma -
Tayade, Nilesh -
Valdis.Kletnieks＠vt.edu