Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build

Chan Kim ckim at etri.re.kr
Wed Apr 27 05:05:08 EDT 2022 

Hi, all

 

I just found that when the linux kernel saves the current x29 and x30 at the new stack bottom, (it usually does that when entering a function)

The stored x30 value (lr register) has it top 16bits altered to some strange value. So if I fix those top 16bits to 0xffff in the stack, and if I repeat fixing this stored x30 values, I can see more backtrace information. I filed a bug report to bug.linaro.org. This strange thing doesn’t happen with linux version < 5.10 to me so it doesn’t look like qemu bug. 

 

Chan Kim

 

From: Chan Kim <ckim at etri.re.kr> 
Sent: Tuesday, April 26, 2022 10:47 AM
To: 'qemu-devel' <qemu-devel at nongnu.org>; 'kernelnewbies' <kernelnewbies at kernelnewbies.org>
Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build

 

Hello,

I hope somebody could check this case. It’s easily reproducible for anybody working with qemu and arm64 linux.

I returned to this problem and made another observation.(showing the back-trace is really broken).

 

In another incident of breakpoint at function __driver_attach, (right after it stopped at the function)

 

    (gdb) bt

    #0  __driver_attach (dev=dev at entry=0xffff0000401d1810, data=data at entry=0xffff800011bbbbb8 <mxc_gpio_driver+40>) at drivers/base/dd.c:1046

    #1  0xffff8000107684f8 in bus_for_each_dev (bus=0xffff800011cba910 <platform_bus_type>, start=0x0, data=0xffff800011bbbbb8 <mxc_gpio_driver+40>, fn=0xffff80001076b860 <__driver_attach>) at drivers/base/bus.c:307

    #2  0xb8cd80001076a594 in ?? ()

    Backtrace stopped: previous frame identical to this frame (corrupt stack?)

 

   (gdb) x/5g $sp

0xffff800011dcbcc0: 0xffff800011dcbd20  0xb8cd80001076a594

0xffff800011dcbcd0: 0xffff80001076b860  0xffff800011bbbbb8

0xffff800011dcbce0: 0x0000000000000000

 

Because it's right after the pc reached the function __driver_attach, the sp is still not updated from previous function (bus_for_each_dev). And the first two values at the $sp are supposed to be the fp and lr of the previous function (see https://stackoverflow.com/questions/66098678/understanding-aarch64-assembly-function-call-how-is-stack-operated arm64 stores previous function's fp and lr at the bottom of new stack frame as it enters a function). The lr (link register, the address to return after this bus_for_each_dev function) is really 0xb8cd80001076a594 which is weird (not a kernel address). The following 3 values are function arguments for bus_for_each_dev and they look correct.

 

From: Chan Kim <ckim at etri.re.kr <mailto:ckim at etri.re.kr> > 
Sent: Saturday, April 23, 2022 8:37 AM
To: 'Mulyadi Santosa' <mulyadi.santosa at gmail.com <mailto:mulyadi.santosa at gmail.com> >
Cc: 'qemu-devel' <qemu-devel at nongnu.org <mailto:qemu-devel at nongnu.org> >; 'kernelnewbies' <kernelnewbies at kernelnewbies.org <mailto:kernelnewbies at kernelnewbies.org> >
Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build

 

Hi, Mulyadi

Thank you for replying.

I found CONFIG_DEBUG_FRAME_POINTER, CONFIG_DEBUG_INFO are already set by default.

And I tried adding CONFIG_DEBUG_KERNEL, CONFIG_KGDB, CONFIG_GDB_SCRIPTS, CONFIG_STACKTRACE all to no avail.

Regards,

Chan

 

From: Mulyadi Santosa <mulyadi.santosa at gmail.com <mailto:mulyadi.santosa at gmail.com> > 
Sent: Friday, April 22, 2022 11:53 PM
To: Chan Kim <ckim at etri.re.kr <mailto:ckim at etri.re.kr> >
Cc: qemu-devel <qemu-devel at nongnu.org <mailto:qemu-devel at nongnu.org> >; kernelnewbies <kernelnewbies at kernelnewbies.org <mailto:kernelnewbies at kernelnewbies.org> >
Subject: Re: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build

 

 

 

On Fri, Apr 22, 2022 at 7:30 PM Chan Kim <ckim at etri.re.kr <mailto:ckim at etri.re.kr> > wrote:

Hello all,

Really strange thing happening here.. I can't see the full stack trace with
'bt' command in gdb.
So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's
happening too!
(it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But
it's not happening with linux-5.4.21)

I would be grateful if somebody could check if this happens to other people
or just me.

1. download linux-5.1.122 tarball from https://www.kernel.org/ 
2. uncompress it and set env variable ARCH=arm64,
CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc`
Image"
3. download qemu-6.2.0 from https://www.qemu.org/
4. uncompress it and do "mkdir build" "cd build" "../configure
--target-list=aarch64-softmmu --enable-debug"
5. run qemu and wait for debugger to attach.
qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine
virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel
linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device
virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram
init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S
6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x\

 

It has been long time since I compiled linux kernel but I guess, either you need to compile kernel with enabled frame pointer, and/or you need to enable debug symbol embedded into final kernel image. cmiiw

 

gdb_script"
(gdb_script content : 
target remote :1234
layout src
b start_kernel
b __driver_attach
)

Now, in gdb, when you press 'c' twice, it'll stop at the first
__driver_attach. (first one stops at start_kernel).
When you are at __attach_driver, type 'bt'. See if you see the full function
stack trace.
This is what I see. 
(gdb) bt
#0  __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358
<dummy_regulator_driver+40>)
    at drivers/base/dd.c:1060
#1  0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>,
start=<optimized out>,
    data=0xffff800011dc2358 <dummy_regulator_driver+40>,
fn=0xffff8000107a6f60 <__driver_attach>)
    at drivers/base/bus.c:305
#2  0xd6d78000107a5c58 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

I used to see more thatn 20 stacks trace but strangely I see only two. 
I can still see many stacks for linux-5.4.21 that I was working with in the
past. 
Could anyone check if this happens to anyone?
I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build,
the kernel will boot ok to the shell prompt.
Only the gdb can't show the stack levels.

My OS : ubuntu-20.04  5.13.0-35-generic

$ aarch64-none-elf-gdb --version
GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11
(arm-10.16)) 10.1.90.20201028-git
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Thank you.
Chan Kim





_______________________________________________
Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.org <mailto:Kernelnewbies at kernelnewbies.org> 
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies



-- 

regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com <http://the-hydra.blogspot.com> 
training: mulyaditraining.blogspot.com <http://mulyaditraining.blogspot.com> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20220427/f4c94f92/attachment-0001.html>

More information about the Kernelnewbies mailing list