Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?
Hello all, As the title says, any of you have successfully tried to debug Linux 4.13 with QEMU? My problem is that it is not possible to break (even with hbreak) in any function of the kernel, from the most used (schedule or spin_lock) to the most obvious (uptime_proc_show when /proc/uptime is read). To be more specific, QEMU correctly stops if I put "-s -S" on the command line, and perfectly continue the execution of the kernel when I connect gdb using 'target remote :1234'. However, any breakpoint is simply ignored. I have followed any possible tutorial and any possible answer to similar user questions. I also tried to do the same on a 16.04 version of Ubuntu, but the result is always the same. In particular, I have followed https://stackoverflow.com/questions/11408041/how-to- debug-the-linux-kernel-with-gdb-and-qemu, using both a busybox image and a raw filesystem image of Archlinux. There is any known problem in debugging Linux? How can I "debug" this debug process? Thank you! -- Stay open, be free.
2017-09-20 4:18 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
Hello all,
As the title says, any of you have successfully tried to debug Linux 4.13 with QEMU? My problem is that it is not possible to break (even with hbreak) in any function of the kernel, from the most used (schedule or spin_lock) to the most obvious (uptime_proc_show when /proc/uptime is read).
To be more specific, QEMU correctly stops if I put "-s -S" on the command line, and perfectly continue the execution of the kernel when I connect gdb using 'target remote :1234'. However, any breakpoint is simply ignored.
I encountered this problem in my debian testing. Any "break" or "hbreak" point is not triggered even if I set breakpoint at "start_kernel". The version of those software are as follows : qemu 1:2.8+dfsg-7 amd64 gdb 7.12-6 amd64 kernel 4.13.0 (build by myself) But I found one interesting phenomenon: If you try to "Ctrl + C" to stop the gdb when you see busybox is already working, you will see one special error: Remote 'g' packet reply is too long: 000000000000000017d11000008ef4810120008000000000fdfb8b07000000000d352828000000004040010000000000903fe081ffffffff883fe081ffffffff00000000000e0000ffffffffffe0ffffffffffff07ffffffffffffffff9fffff17d11000008ef4810000000000800000fffffffff8ffffffffff0000ffffffff2ddbf481ffffffff4600000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000 I try to use busybox and one simple "helloworld" rootfs. The result shows it is not related with rootfs. It seems gdb does not handle register length of amd64 due to https://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-w.... But until now, I did not figure out the reason why this error occurs and how to fix it. -- My best regards to you. No System Is Safe! Dongliang Mu
I have followed any possible tutorial and any possible answer to similar user questions. I also tried to do the same on a 16.04 version of Ubuntu, but the result is always the same. In particular, I have followed https://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-w..., using both a busybox image and a raw filesystem image of Archlinux.
There is any known problem in debugging Linux? How can I "debug" this debug process?
Thank you!
-- Stay open, be free.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On 20/09/17 at 11:20am, 慕冬亮 wrote:
2017-09-20 4:18 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
Hello all,
As the title says, any of you have successfully tried to debug Linux 4.13 with QEMU?
I encountered this problem in my debian testing. Any "break" or "hbreak" point is not triggered even if I set breakpoint at "start_kernel". [cut]
So, after two weeks of a headache I took the courage to send an email here, and after only 3 hours after the email (!!!) I found the cause of the problem. Without being too long, the problem is KALSR. You have to disable it disabling the kernel option "Randomize the address of the kernel image (KASLR)", inside Processor type and features. In this way, gdb can translate all the symbols correctly (symbols <-> addresses). I really don't know how the developers test their code... because there is no evidence for this.
But I found one interesting phenomenon:
If you try to "Ctrl + C" to stop the gdb when you see busybox is already working, you will see one special error:
Remote 'g' packet reply is too long: [cut]
The answer to this issue is found easily on the internet. You have to stop using -S, because it creates problems. Please take a look to [1]. Thanks for your reply and have a nice day! JJD [1] https://www.google.com/search?q=packet+reply+too+long+gdb+linux
On Wed, Sep 20, 2017 at 09:29:17PM +0200, jjDaNiMoTh wrote:
On 20/09/17 at 11:20am, 慕冬亮 wrote:
2017-09-20 4:18 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
Hello all,
As the title says, any of you have successfully tried to debug Linux 4.13 with QEMU?
I encountered this problem in my debian testing. Any "break" or "hbreak" point is not triggered even if I set breakpoint at "start_kernel". [cut]
So, after two weeks of a headache I took the courage to send an email here, and after only 3 hours after the email (!!!) I found the cause of the problem.
Without being too long, the problem is KALSR. You have to disable it disabling the kernel option "Randomize the address of the kernel image (KASLR)", inside Processor type and features. In this way, gdb can translate all the symbols correctly (symbols <-> addresses).
I really don't know how the developers test their code... because there is no evidence for this.
Why do you think that it requires developers to use a debugger and qemu to test their code? I've never used a debugger on the kernel in all of my years of kernel development... Glad you found a solution and posted it. greg k-h
Hi,
Without being too long, the problem is KALSR. You have to disable it disabling the kernel option "Randomize the address of the kernel image (KASLR)", inside Processor type and features. In this way, gdb can translate all the symbols correctly (symbols <-> addresses).
I really don't know how the developers test their code... because there is no evidence for this.
Why do you think that it requires developers to use a debugger and qemu to test their code? I've never used a debugger on the kernel in all of my years of kernel development...
I agree the developers should not be doing it. It is documented that gdb/kgdb will not work if KALSAR is enabled. Please refer - https://github.com/torvalds/linux/commit/e604f1cb85367d2e5fd4cf253296d190996... Regards, Saket Sinha
2017-09-21 9:21 GMT+02:00 Greg KH <greg@kroah.com>:
Why do you think that it requires developers to use a debugger and qemu to test their code? I've never used a debugger on the kernel in all of my years of kernel development...
I imagined that, at least in early development phases, developers can benefit from the use of a debugger, but it seems that I was wrong. Of course, for newbies like me, a debugger is essential to understand flow and relations between kernel code, and I'm happy that this solution is already in the documentation (unfortunately, not for the version I was using, 4.13). Maybe the patch pointed out by Saket can be backported to releases which have the possibility to enable KALSR, such as 4.9 (LTS) and 4.13 itself (stable)? [1] https://www.kernel.org/doc/html/v4.13/dev-tools/gdb-kernel-debugging.html JJD -- Stay open, be free.
On Thu, Sep 21, 2017 at 10:43:08AM +0200, jjDaNiMoTh wrote:
2017-09-21 9:21 GMT+02:00 Greg KH <greg@kroah.com>:
Why do you think that it requires developers to use a debugger and qemu to test their code? I've never used a debugger on the kernel in all of my years of kernel development...
I imagined that, at least in early development phases, developers can benefit from the use of a debugger, but it seems that I was wrong. Of course, for newbies like me, a debugger is essential to understand flow and relations between kernel code, and I'm happy that this solution is already in the documentation (unfortunately, not for the version I was using, 4.13). Maybe the patch pointed out by Saket can be backported to releases which have the possibility to enable KALSR, such as 4.9 (LTS) and 4.13 itself (stable)?
Sure, you can request it, look at the documentation here for how to do that: https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
2017-09-20 15:29 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
On 20/09/17 at 11:20am, 慕冬亮 wrote:
2017-09-20 4:18 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
Hello all,
As the title says, any of you have successfully tried to debug Linux 4.13 with QEMU?
I encountered this problem in my debian testing. Any "break" or "hbreak" point is not triggered even if I set breakpoint at "start_kernel". [cut]
So, after two weeks of a headache I took the courage to send an email here, and after only 3 hours after the email (!!!) I found the cause of the problem.
Without being too long, the problem is KALSR. You have to disable it disabling the kernel option "Randomize the address of the kernel image (KASLR)", inside Processor type and features. In this way, gdb can translate all the symbols correctly (symbols <-> addresses).
I really don't know how the developers test their code... because there is no evidence for this.
But I found one interesting phenomenon:
If you try to "Ctrl + C" to stop the gdb when you see busybox is already working, you will see one special error:
Remote 'g' packet reply is too long: [cut]
The answer to this issue is found easily on the internet. You have to stop using -S, because it creates problems. Please take a look to [1].
I have seen this solution during searching answers. But without "-S" (-S stalls QEMU CPU at startup), the kernel will not stop util it reaches busybox command line. In other word, you have no method to debug the Linux early boot process. So I try to find solution to fix it and debug the kernel from start_kernel to review the early boot process of linux Kernel. I do not know how one senior developer debugs kernel code. I just want to use qemu, gdb to debug and learn how kernel works.
Thanks for your reply and have a nice day!
JJD
[1] https://www.google.com/search?q=packet+reply+too+long+gdb+linux
Hi,
I have seen this solution during searching answers. But without "-S" (-S stalls QEMU CPU at startup), the kernel will not stop util it reaches busybox command line. In other word, you have no method to debug the Linux early boot process.
The early boot process is highly architecture dependent. You can emulate a virtual device close to what you have as a real device and write drivers for that. Please follow the below link as to how Android emulator(which uses qemu at backend) does it - https://android.googlesource.com/platform/external/qemu/+/master/docs/GOLDFI... Regards, Saket Sinha
participants (4)
-
Greg KH -
jjDaNiMoTh -
Saket Sinha -
慕冬亮