Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build
Hello all, Really strange thing happening here.. I can't see the full stack trace with 'bt' command in gdb. So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's happening too! (it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But it's not happening with linux-5.4.21) I would be grateful if somebody could check if this happens to other people or just me. 1. download linux-5.1.122 tarball from https://www.kernel.org/ 2. uncompress it and set env variable ARCH=arm64, CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc` Image" 3. download qemu-6.2.0 from https://www.qemu.org/ 4. uncompress it and do "mkdir build" "cd build" "../configure --target-list=aarch64-softmmu --enable-debug" 5. run qemu and wait for debugger to attach. qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S 6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x gdb_script" (gdb_script content : target remote :1234 layout src b start_kernel b __driver_attach ) Now, in gdb, when you press 'c' twice, it'll stop at the first __driver_attach. (first one stops at start_kernel). When you are at __attach_driver, type 'bt'. See if you see the full function stack trace. This is what I see. (gdb) bt #0 __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358 <dummy_regulator_driver+40>) at drivers/base/dd.c:1060 #1 0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>, start=<optimized out>, data=0xffff800011dc2358 <dummy_regulator_driver+40>, fn=0xffff8000107a6f60 <__driver_attach>) at drivers/base/bus.c:305 #2 0xd6d78000107a5c58 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) I used to see more thatn 20 stacks trace but strangely I see only two. I can still see many stacks for linux-5.4.21 that I was working with in the past. Could anyone check if this happens to anyone? I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build, the kernel will boot ok to the shell prompt. Only the gdb can't show the stack levels. My OS : ubuntu-20.04 5.13.0-35-generic $ aarch64-none-elf-gdb --version GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.1.90.20201028-git Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Thank you. Chan Kim
On Fri, Apr 22, 2022 at 7:30 PM Chan Kim <ckim@etri.re.kr> wrote:
Hello all,
Really strange thing happening here.. I can't see the full stack trace with 'bt' command in gdb. So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's happening too! (it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But it's not happening with linux-5.4.21)
I would be grateful if somebody could check if this happens to other people or just me.
1. download linux-5.1.122 tarball from https://www.kernel.org/ 2. uncompress it and set env variable ARCH=arm64, CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc` Image" 3. download qemu-6.2.0 from https://www.qemu.org/ 4. uncompress it and do "mkdir build" "cd build" "../configure --target-list=aarch64-softmmu --enable-debug" 5. run qemu and wait for debugger to attach. qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S 6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x\
It has been long time since I compiled linux kernel but I guess, either you need to compile kernel with enabled frame pointer, and/or you need to enable debug symbol embedded into final kernel image. cmiiw
gdb_script" (gdb_script content : target remote :1234 layout src b start_kernel b __driver_attach )
Now, in gdb, when you press 'c' twice, it'll stop at the first __driver_attach. (first one stops at start_kernel). When you are at __attach_driver, type 'bt'. See if you see the full function stack trace. This is what I see. (gdb) bt #0 __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358 <dummy_regulator_driver+40>) at drivers/base/dd.c:1060 #1 0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>, start=<optimized out>, data=0xffff800011dc2358 <dummy_regulator_driver+40>, fn=0xffff8000107a6f60 <__driver_attach>) at drivers/base/bus.c:305 #2 0xd6d78000107a5c58 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?)
I used to see more thatn 20 stacks trace but strangely I see only two. I can still see many stacks for linux-5.4.21 that I was working with in the past. Could anyone check if this happens to anyone? I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build, the kernel will boot ok to the shell prompt. Only the gdb can't show the stack levels.
My OS : ubuntu-20.04 5.13.0-35-generic
$ aarch64-none-elf-gdb --version GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.1.90.20201028-git Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Thank you. Chan Kim
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Hi, Mulyadi Thank you for replying. I found CONFIG_DEBUG_FRAME_POINTER, CONFIG_DEBUG_INFO are already set by default. And I tried adding CONFIG_DEBUG_KERNEL, CONFIG_KGDB, CONFIG_GDB_SCRIPTS, CONFIG_STACKTRACE all to no avail. Regards, Chan From: Mulyadi Santosa <mulyadi.santosa@gmail.com> Sent: Friday, April 22, 2022 11:53 PM To: Chan Kim <ckim@etri.re.kr> Cc: qemu-devel <qemu-devel@nongnu.org>; kernelnewbies <kernelnewbies@kernelnewbies.org> Subject: Re: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build On Fri, Apr 22, 2022 at 7:30 PM Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > wrote: Hello all, Really strange thing happening here.. I can't see the full stack trace with 'bt' command in gdb. So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's happening too! (it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But it's not happening with linux-5.4.21) I would be grateful if somebody could check if this happens to other people or just me. 1. download linux-5.1.122 tarball from https://www.kernel.org/ 2. uncompress it and set env variable ARCH=arm64, CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc` Image" 3. download qemu-6.2.0 from https://www.qemu.org/ 4. uncompress it and do "mkdir build" "cd build" "../configure --target-list=aarch64-softmmu --enable-debug" 5. run qemu and wait for debugger to attach. qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S 6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x\ It has been long time since I compiled linux kernel but I guess, either you need to compile kernel with enabled frame pointer, and/or you need to enable debug symbol embedded into final kernel image. cmiiw gdb_script" (gdb_script content : target remote :1234 layout src b start_kernel b __driver_attach ) Now, in gdb, when you press 'c' twice, it'll stop at the first __driver_attach. (first one stops at start_kernel). When you are at __attach_driver, type 'bt'. See if you see the full function stack trace. This is what I see. (gdb) bt #0 __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358 <dummy_regulator_driver+40>) at drivers/base/dd.c:1060 #1 0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>, start=<optimized out>, data=0xffff800011dc2358 <dummy_regulator_driver+40>, fn=0xffff8000107a6f60 <__driver_attach>) at drivers/base/bus.c:305 #2 0xd6d78000107a5c58 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) I used to see more thatn 20 stacks trace but strangely I see only two. I can still see many stacks for linux-5.4.21 that I was working with in the past. Could anyone check if this happens to anyone? I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build, the kernel will boot ok to the shell prompt. Only the gdb can't show the stack levels. My OS : ubuntu-20.04 5.13.0-35-generic $ aarch64-none-elf-gdb --version GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.1.90.20201028-git Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Thank you. Chan Kim _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org <mailto:Kernelnewbies@kernelnewbies.org> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com <http://the-hydra.blogspot.com> training: mulyaditraining.blogspot.com <http://mulyaditraining.blogspot.com>
Hello, I hope somebody could check this case. It’s easily reproducible for anybody working with qemu and arm64 linux. I returned to this problem and made another observation.(showing the back-trace is really broken). In another incident of breakpoint at function __driver_attach, (right after it stopped at the function) (gdb) bt #0 __driver_attach (dev=dev@entry=0xffff0000401d1810, data=data@entry=0xffff800011bbbbb8 <mxc_gpio_driver+40>) at drivers/base/dd.c:1046 #1 0xffff8000107684f8 in bus_for_each_dev (bus=0xffff800011cba910 <platform_bus_type>, start=0x0, data=0xffff800011bbbbb8 <mxc_gpio_driver+40>, fn=0xffff80001076b860 <__driver_attach>) at drivers/base/bus.c:307 #2 0xb8cd80001076a594 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) x/5g $sp 0xffff800011dcbcc0: 0xffff800011dcbd20 0xb8cd80001076a594 0xffff800011dcbcd0: 0xffff80001076b860 0xffff800011bbbbb8 0xffff800011dcbce0: 0x0000000000000000 Because it's right after the pc reached the function __driver_attach, the sp is still not updated from previous function (bus_for_each_dev). And the first two values at the $sp are supposed to be the fp and lr of the previous function (see https://stackoverflow.com/questions/66098678/understanding-aarch64-assembly-... arm64 stores previous function's fp and lr at the bottom of new stack frame as it enters a function). The lr (link register, the address to return after this bus_for_each_dev function) is really 0xb8cd80001076a594 which is weird (not a kernel address). The following 3 values are function arguments for bus_for_each_dev and they look correct. From: Chan Kim <ckim@etri.re.kr> Sent: Saturday, April 23, 2022 8:37 AM To: 'Mulyadi Santosa' <mulyadi.santosa@gmail.com> Cc: 'qemu-devel' <qemu-devel@nongnu.org>; 'kernelnewbies' <kernelnewbies@kernelnewbies.org> Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Hi, Mulyadi Thank you for replying. I found CONFIG_DEBUG_FRAME_POINTER, CONFIG_DEBUG_INFO are already set by default. And I tried adding CONFIG_DEBUG_KERNEL, CONFIG_KGDB, CONFIG_GDB_SCRIPTS, CONFIG_STACKTRACE all to no avail. Regards, Chan From: Mulyadi Santosa <mulyadi.santosa@gmail.com <mailto:mulyadi.santosa@gmail.com> > Sent: Friday, April 22, 2022 11:53 PM To: Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > Cc: qemu-devel <qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org> >; kernelnewbies <kernelnewbies@kernelnewbies.org <mailto:kernelnewbies@kernelnewbies.org> > Subject: Re: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build On Fri, Apr 22, 2022 at 7:30 PM Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > wrote: Hello all, Really strange thing happening here.. I can't see the full stack trace with 'bt' command in gdb. So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's happening too! (it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But it's not happening with linux-5.4.21) I would be grateful if somebody could check if this happens to other people or just me. 1. download linux-5.1.122 tarball from https://www.kernel.org/ 2. uncompress it and set env variable ARCH=arm64, CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc` Image" 3. download qemu-6.2.0 from https://www.qemu.org/ 4. uncompress it and do "mkdir build" "cd build" "../configure --target-list=aarch64-softmmu --enable-debug" 5. run qemu and wait for debugger to attach. qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S 6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x\ It has been long time since I compiled linux kernel but I guess, either you need to compile kernel with enabled frame pointer, and/or you need to enable debug symbol embedded into final kernel image. cmiiw gdb_script" (gdb_script content : target remote :1234 layout src b start_kernel b __driver_attach ) Now, in gdb, when you press 'c' twice, it'll stop at the first __driver_attach. (first one stops at start_kernel). When you are at __attach_driver, type 'bt'. See if you see the full function stack trace. This is what I see. (gdb) bt #0 __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358 <dummy_regulator_driver+40>) at drivers/base/dd.c:1060 #1 0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>, start=<optimized out>, data=0xffff800011dc2358 <dummy_regulator_driver+40>, fn=0xffff8000107a6f60 <__driver_attach>) at drivers/base/bus.c:305 #2 0xd6d78000107a5c58 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) I used to see more thatn 20 stacks trace but strangely I see only two. I can still see many stacks for linux-5.4.21 that I was working with in the past. Could anyone check if this happens to anyone? I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build, the kernel will boot ok to the shell prompt. Only the gdb can't show the stack levels. My OS : ubuntu-20.04 5.13.0-35-generic $ aarch64-none-elf-gdb --version GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.1.90.20201028-git Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Thank you. Chan Kim _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org <mailto:Kernelnewbies@kernelnewbies.org> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com <http://the-hydra.blogspot.com> training: mulyaditraining.blogspot.com <http://mulyaditraining.blogspot.com>
Hi, all I just found that when the linux kernel saves the current x29 and x30 at the new stack bottom, (it usually does that when entering a function) The stored x30 value (lr register) has it top 16bits altered to some strange value. So if I fix those top 16bits to 0xffff in the stack, and if I repeat fixing this stored x30 values, I can see more backtrace information. I filed a bug report to bug.linaro.org. This strange thing doesn’t happen with linux version < 5.10 to me so it doesn’t look like qemu bug. Chan Kim From: Chan Kim <ckim@etri.re.kr> Sent: Tuesday, April 26, 2022 10:47 AM To: 'qemu-devel' <qemu-devel@nongnu.org>; 'kernelnewbies' <kernelnewbies@kernelnewbies.org> Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Hello, I hope somebody could check this case. It’s easily reproducible for anybody working with qemu and arm64 linux. I returned to this problem and made another observation.(showing the back-trace is really broken). In another incident of breakpoint at function __driver_attach, (right after it stopped at the function) (gdb) bt #0 __driver_attach (dev=dev@entry=0xffff0000401d1810, data=data@entry=0xffff800011bbbbb8 <mxc_gpio_driver+40>) at drivers/base/dd.c:1046 #1 0xffff8000107684f8 in bus_for_each_dev (bus=0xffff800011cba910 <platform_bus_type>, start=0x0, data=0xffff800011bbbbb8 <mxc_gpio_driver+40>, fn=0xffff80001076b860 <__driver_attach>) at drivers/base/bus.c:307 #2 0xb8cd80001076a594 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) x/5g $sp 0xffff800011dcbcc0: 0xffff800011dcbd20 0xb8cd80001076a594 0xffff800011dcbcd0: 0xffff80001076b860 0xffff800011bbbbb8 0xffff800011dcbce0: 0x0000000000000000 Because it's right after the pc reached the function __driver_attach, the sp is still not updated from previous function (bus_for_each_dev). And the first two values at the $sp are supposed to be the fp and lr of the previous function (see https://stackoverflow.com/questions/66098678/understanding-aarch64-assembly-... arm64 stores previous function's fp and lr at the bottom of new stack frame as it enters a function). The lr (link register, the address to return after this bus_for_each_dev function) is really 0xb8cd80001076a594 which is weird (not a kernel address). The following 3 values are function arguments for bus_for_each_dev and they look correct. From: Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > Sent: Saturday, April 23, 2022 8:37 AM To: 'Mulyadi Santosa' <mulyadi.santosa@gmail.com <mailto:mulyadi.santosa@gmail.com> > Cc: 'qemu-devel' <qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org> >; 'kernelnewbies' <kernelnewbies@kernelnewbies.org <mailto:kernelnewbies@kernelnewbies.org> > Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Hi, Mulyadi Thank you for replying. I found CONFIG_DEBUG_FRAME_POINTER, CONFIG_DEBUG_INFO are already set by default. And I tried adding CONFIG_DEBUG_KERNEL, CONFIG_KGDB, CONFIG_GDB_SCRIPTS, CONFIG_STACKTRACE all to no avail. Regards, Chan From: Mulyadi Santosa <mulyadi.santosa@gmail.com <mailto:mulyadi.santosa@gmail.com> > Sent: Friday, April 22, 2022 11:53 PM To: Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > Cc: qemu-devel <qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org> >; kernelnewbies <kernelnewbies@kernelnewbies.org <mailto:kernelnewbies@kernelnewbies.org> > Subject: Re: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build On Fri, Apr 22, 2022 at 7:30 PM Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > wrote: Hello all, Really strange thing happening here.. I can't see the full stack trace with 'bt' command in gdb. So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's happening too! (it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But it's not happening with linux-5.4.21) I would be grateful if somebody could check if this happens to other people or just me. 1. download linux-5.1.122 tarball from https://www.kernel.org/ 2. uncompress it and set env variable ARCH=arm64, CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc` Image" 3. download qemu-6.2.0 from https://www.qemu.org/ 4. uncompress it and do "mkdir build" "cd build" "../configure --target-list=aarch64-softmmu --enable-debug" 5. run qemu and wait for debugger to attach. qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S 6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x\ It has been long time since I compiled linux kernel but I guess, either you need to compile kernel with enabled frame pointer, and/or you need to enable debug symbol embedded into final kernel image. cmiiw gdb_script" (gdb_script content : target remote :1234 layout src b start_kernel b __driver_attach ) Now, in gdb, when you press 'c' twice, it'll stop at the first __driver_attach. (first one stops at start_kernel). When you are at __attach_driver, type 'bt'. See if you see the full function stack trace. This is what I see. (gdb) bt #0 __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358 <dummy_regulator_driver+40>) at drivers/base/dd.c:1060 #1 0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>, start=<optimized out>, data=0xffff800011dc2358 <dummy_regulator_driver+40>, fn=0xffff8000107a6f60 <__driver_attach>) at drivers/base/bus.c:305 #2 0xd6d78000107a5c58 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) I used to see more thatn 20 stacks trace but strangely I see only two. I can still see many stacks for linux-5.4.21 that I was working with in the past. Could anyone check if this happens to anyone? I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build, the kernel will boot ok to the shell prompt. Only the gdb can't show the stack levels. My OS : ubuntu-20.04 5.13.0-35-generic $ aarch64-none-elf-gdb --version GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.1.90.20201028-git Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Thank you. Chan Kim _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org <mailto:Kernelnewbies@kernelnewbies.org> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com <http://the-hydra.blogspot.com> training: mulyaditraining.blogspot.com <http://mulyaditraining.blogspot.com>
Hi folks, Sorry for many emails. I just found out by turning CONFIG_ARM64_PTR_AUTH off in armv8.3 when building linux, I can avoid this problem. (I noticed the instruction ‘pacia’ in the function assembly code) Thank you. Chan Kim From: Chan Kim <ckim@etri.re.kr> Sent: Wednesday, April 27, 2022 6:05 PM To: 'qemu-devel' <qemu-devel@nongnu.org>; 'kernelnewbies' <kernelnewbies@kernelnewbies.org> Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Hi, all I just found that when the linux kernel saves the current x29 and x30 at the new stack bottom, (it usually does that when entering a function) The stored x30 value (lr register) has it top 16bits altered to some strange value. So if I fix those top 16bits to 0xffff in the stack, and if I repeat fixing this stored x30 values, I can see more backtrace information. I filed a bug report to bug.linaro.org. This strange thing doesn’t happen with linux version < 5.10 to me so it doesn’t look like qemu bug. Chan Kim From: Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > Sent: Tuesday, April 26, 2022 10:47 AM To: 'qemu-devel' <qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org> >; 'kernelnewbies' <kernelnewbies@kernelnewbies.org <mailto:kernelnewbies@kernelnewbies.org> > Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Hello, I hope somebody could check this case. It’s easily reproducible for anybody working with qemu and arm64 linux. I returned to this problem and made another observation.(showing the back-trace is really broken). In another incident of breakpoint at function __driver_attach, (right after it stopped at the function) (gdb) bt #0 __driver_attach (dev=dev@entry=0xffff0000401d1810, data=data@entry=0xffff800011bbbbb8 <mxc_gpio_driver+40>) at drivers/base/dd.c:1046 #1 0xffff8000107684f8 in bus_for_each_dev (bus=0xffff800011cba910 <platform_bus_type>, start=0x0, data=0xffff800011bbbbb8 <mxc_gpio_driver+40>, fn=0xffff80001076b860 <__driver_attach>) at drivers/base/bus.c:307 #2 0xb8cd80001076a594 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) x/5g $sp 0xffff800011dcbcc0: 0xffff800011dcbd20 0xb8cd80001076a594 0xffff800011dcbcd0: 0xffff80001076b860 0xffff800011bbbbb8 0xffff800011dcbce0: 0x0000000000000000 Because it's right after the pc reached the function __driver_attach, the sp is still not updated from previous function (bus_for_each_dev). And the first two values at the $sp are supposed to be the fp and lr of the previous function (see https://stackoverflow.com/questions/66098678/understanding-aarch64-assembly-... arm64 stores previous function's fp and lr at the bottom of new stack frame as it enters a function). The lr (link register, the address to return after this bus_for_each_dev function) is really 0xb8cd80001076a594 which is weird (not a kernel address). The following 3 values are function arguments for bus_for_each_dev and they look correct. From: Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > Sent: Saturday, April 23, 2022 8:37 AM To: 'Mulyadi Santosa' <mulyadi.santosa@gmail.com <mailto:mulyadi.santosa@gmail.com> > Cc: 'qemu-devel' <qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org> >; 'kernelnewbies' <kernelnewbies@kernelnewbies.org <mailto:kernelnewbies@kernelnewbies.org> > Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Hi, Mulyadi Thank you for replying. I found CONFIG_DEBUG_FRAME_POINTER, CONFIG_DEBUG_INFO are already set by default. And I tried adding CONFIG_DEBUG_KERNEL, CONFIG_KGDB, CONFIG_GDB_SCRIPTS, CONFIG_STACKTRACE all to no avail. Regards, Chan From: Mulyadi Santosa <mulyadi.santosa@gmail.com <mailto:mulyadi.santosa@gmail.com> > Sent: Friday, April 22, 2022 11:53 PM To: Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > Cc: qemu-devel <qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org> >; kernelnewbies <kernelnewbies@kernelnewbies.org <mailto:kernelnewbies@kernelnewbies.org> > Subject: Re: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build On Fri, Apr 22, 2022 at 7:30 PM Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > wrote: Hello all, Really strange thing happening here.. I can't see the full stack trace with 'bt' command in gdb. So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's happening too! (it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But it's not happening with linux-5.4.21) I would be grateful if somebody could check if this happens to other people or just me. 1. download linux-5.1.122 tarball from https://www.kernel.org/ 2. uncompress it and set env variable ARCH=arm64, CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc` Image" 3. download qemu-6.2.0 from https://www.qemu.org/ 4. uncompress it and do "mkdir build" "cd build" "../configure --target-list=aarch64-softmmu --enable-debug" 5. run qemu and wait for debugger to attach. qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S 6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x\ It has been long time since I compiled linux kernel but I guess, either you need to compile kernel with enabled frame pointer, and/or you need to enable debug symbol embedded into final kernel image. cmiiw gdb_script" (gdb_script content : target remote :1234 layout src b start_kernel b __driver_attach ) Now, in gdb, when you press 'c' twice, it'll stop at the first __driver_attach. (first one stops at start_kernel). When you are at __attach_driver, type 'bt'. See if you see the full function stack trace. This is what I see. (gdb) bt #0 __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358 <dummy_regulator_driver+40>) at drivers/base/dd.c:1060 #1 0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>, start=<optimized out>, data=0xffff800011dc2358 <dummy_regulator_driver+40>, fn=0xffff8000107a6f60 <__driver_attach>) at drivers/base/bus.c:305 #2 0xd6d78000107a5c58 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) I used to see more thatn 20 stacks trace but strangely I see only two. I can still see many stacks for linux-5.4.21 that I was working with in the past. Could anyone check if this happens to anyone? I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build, the kernel will boot ok to the shell prompt. Only the gdb can't show the stack levels. My OS : ubuntu-20.04 5.13.0-35-generic $ aarch64-none-elf-gdb --version GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.1.90.20201028-git Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Thank you. Chan Kim _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org <mailto:Kernelnewbies@kernelnewbies.org> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com <http://the-hydra.blogspot.com> training: mulyaditraining.blogspot.com <http://mulyaditraining.blogspot.com>
participants (2)
-
Chan Kim -
Mulyadi Santosa