<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:굴림;
panose-1:2 11 6 0 0 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"맑은 고딕";
panose-1:2 11 5 3 2 0 0 2 0 4;}
@font-face
{font-family:"\@맑은 고딕";
panose-1:2 11 5 3 2 0 0 2 0 4;}
@font-face
{font-family:"\@굴림";
panose-1:2 11 6 0 0 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:굴림;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:굴림;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"맑은 고딕";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"맑은 고딕";
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"맑은 고딕";
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"맑은 고딕";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:3.0cm 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=KO link=blue vlink=purple><div class=WordSection1><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Hi folks,<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Sorry for many emails.<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>I just found out by turning CONFIG_ARM64_PTR_AUTH off in armv8.3 when building linux, I can avoid this problem.<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>(I noticed the instruction ‘pacia’ in the function assembly code)<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Thank you.<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Chan Kim<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Chan Kim <ckim@etri.re.kr> <br><b>Sent:</b> Wednesday, April 27, 2022 6:05 PM<br><b>To:</b> 'qemu-devel' <qemu-devel@nongnu.org>; 'kernelnewbies' <kernelnewbies@kernelnewbies.org><br><b>Subject:</b> RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build<o:p></o:p></span></p></div></div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Hi, all<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>I just found that when the linux kernel saves the current x29 and x30 at the new stack bottom, (it usually does that when entering a function)<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>The stored x30 value (lr register) has it top 16bits altered to some strange value. So if I fix those top 16bits to 0xffff in the stack, and if I repeat fixing this stored x30 values, I can see more backtrace information. I filed a bug report to bug.linaro.org. This strange thing doesn</span><span style='font-size:10.0pt;font-family:"맑은 고딕"'>’<span lang=EN-US>t happen with linux version < 5.10 to me so it doesn</span>’<span lang=EN-US>t look like qemu bug. <o:p></o:p></span></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Chan Kim<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Chan Kim <<a href="mailto:ckim@etri.re.kr">ckim@etri.re.kr</a>> <br><b>Sent:</b> Tuesday, April 26, 2022 10:47 AM<br><b>To:</b> 'qemu-devel' <<a href="mailto:qemu-devel@nongnu.org">qemu-devel@nongnu.org</a>>; 'kernelnewbies' <<a href="mailto:kernelnewbies@kernelnewbies.org">kernelnewbies@kernelnewbies.org</a>><br><b>Subject:</b> RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build<o:p></o:p></span></p></div></div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Hello,<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>I hope somebody could check this case. It</span><span style='font-size:10.0pt;font-family:"맑은 고딕"'>’<span lang=EN-US>s easily reproducible for anybody working with qemu and arm64 linux.<o:p></o:p></span></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>I returned to this problem and made another observation.(showing the back-trace is really broken).<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>In another incident of breakpoint at function __driver_attach, (right after it stopped at the function)<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'> (gdb) bt<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'> #0 __driver_attach (dev=dev@entry=0xffff0000401d1810, data=data@entry=0xffff800011bbbbb8 <mxc_gpio_driver+40>) at drivers/base/dd.c:1046<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'> #1 0xffff8000107684f8 in bus_for_each_dev (bus=0xffff800011cba910 <platform_bus_type>, start=0x0, data=0xffff800011bbbbb8 <mxc_gpio_driver+40>, fn=0xffff80001076b860 <__driver_attach>) at drivers/base/bus.c:307<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'> #2 0xb8cd80001076a594 in ?? ()<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'> Backtrace stopped: previous frame identical to this frame (corrupt stack?)<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'> (gdb) x/5g $sp<o:p></o:p></span></p><p class=MsoNormal style='text-indent:20.0pt;word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>0xffff800011dcbcc0: 0xffff800011dcbd20 0xb8cd80001076a594<o:p></o:p></span></p><p class=MsoNormal style='text-indent:20.0pt;word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>0xffff800011dcbcd0: 0xffff80001076b860 0xffff800011bbbbb8<o:p></o:p></span></p><p class=MsoNormal style='text-indent:20.0pt;word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>0xffff800011dcbce0: 0x0000000000000000<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Because it's right after the pc reached the function __driver_attach, the sp is still not updated from previous function (bus_for_each_dev). And the first two values at the $sp are supposed to be the fp and lr of the previous function (see <a href="https://stackoverflow.com/questions/66098678/understanding-aarch64-assembly-function-call-how-is-stack-operated">https://stackoverflow.com/questions/66098678/understanding-aarch64-assembly-function-call-how-is-stack-operated</a> arm64 stores previous function's fp and lr at the bottom of new stack frame as it enters a function). The lr (link register, the address to return after this bus_for_each_dev function) is really 0xb8cd80001076a594 which is weird (not a kernel address). The following 3 values are function arguments for bus_for_each_dev and they look correct.<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Chan Kim <<a href="mailto:ckim@etri.re.kr">ckim@etri.re.kr</a>> <br><b>Sent:</b> Saturday, April 23, 2022 8:37 AM<br><b>To:</b> 'Mulyadi Santosa' <<a href="mailto:mulyadi.santosa@gmail.com">mulyadi.santosa@gmail.com</a>><br><b>Cc:</b> 'qemu-devel' <<a href="mailto:qemu-devel@nongnu.org">qemu-devel@nongnu.org</a>>; 'kernelnewbies' <<a href="mailto:kernelnewbies@kernelnewbies.org">kernelnewbies@kernelnewbies.org</a>><br><b>Subject:</b> RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build<o:p></o:p></span></p></div></div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Hi, Mulyadi<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Thank you for replying.<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>I found CONFIG_DEBUG_FRAME_POINTER, CONFIG_DEBUG_INFO are already set by default.<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>And I tried adding CONFIG_DEBUG_KERNEL, CONFIG_KGDB, CONFIG_GDB_SCRIPTS, CONFIG_STACKTRACE all to no avail.<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Regards,<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'>Chan<o:p></o:p></span></p><p class=MsoNormal style='word-break:break-hangul'><span lang=EN-US style='font-size:10.0pt;font-family:"맑은 고딕"'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Mulyadi Santosa <<a href="mailto:mulyadi.santosa@gmail.com">mulyadi.santosa@gmail.com</a>> <br><b>Sent:</b> Friday, April 22, 2022 11:53 PM<br><b>To:</b> Chan Kim <<a href="mailto:ckim@etri.re.kr">ckim@etri.re.kr</a>><br><b>Cc:</b> qemu-devel <<a href="mailto:qemu-devel@nongnu.org">qemu-devel@nongnu.org</a>>; kernelnewbies <<a href="mailto:kernelnewbies@kernelnewbies.org">kernelnewbies@kernelnewbies.org</a>><br><b>Subject:</b> Re: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build<o:p></o:p></span></p></div></div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><div><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><div><div><p class=MsoNormal><span lang=EN-US>On Fri, Apr 22, 2022 at 7:30 PM Chan Kim <<a href="mailto:ckim@etri.re.kr">ckim@etri.re.kr</a>> wrote:<o:p></o:p></span></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><p class=MsoNormal><span lang=EN-US>Hello all,<br><br>Really strange thing happening here.. I can't see the full stack trace with<br>'bt' command in gdb.<br>So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's<br>happening too!<br>(it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But<br>it's not happening with linux-5.4.21)<br><br>I would be grateful if somebody could check if this happens to other people<br>or just me.<br><br>1. download linux-5.1.122 tarball from <a href="https://www.kernel.org/" target="_blank">https://www.kernel.org/</a> <br>2. uncompress it and set env variable ARCH=arm64,<br>CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc`<br>Image"<br>3. download qemu-6.2.0 from <a href="https://www.qemu.org/" target="_blank">https://www.qemu.org/</a><br>4. uncompress it and do "mkdir build" "cd build" "../configure<br>--target-list=aarch64-softmmu --enable-debug"<br>5. run qemu and wait for debugger to attach.<br>qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine<br>virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel<br>linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev<br>user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device<br>virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram<br>init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S<br>6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x\<o:p></o:p></span></p></blockquote><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US>It has been long time since I compiled linux kernel but I guess, either you need to compile kernel with enabled frame pointer, and/or you need to enable debug symbol embedded into final kernel image. cmiiw<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><p class=MsoNormal><span lang=EN-US>gdb_script"<br>(gdb_script content : <br>target remote :1234<br>layout src<br>b start_kernel<br>b __driver_attach<br>)<br><br>Now, in gdb, when you press 'c' twice, it'll stop at the first<br>__driver_attach. (first one stops at start_kernel).<br>When you are at __attach_driver, type 'bt'. See if you see the full function<br>stack trace.<br>This is what I see. <br>(gdb) bt<br>#0 __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358<br><dummy_regulator_driver+40>)<br> at drivers/base/dd.c:1060<br>#1 0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>,<br>start=<optimized out>,<br> data=0xffff800011dc2358 <dummy_regulator_driver+40>,<br>fn=0xffff8000107a6f60 <__driver_attach>)<br> at drivers/base/bus.c:305<br>#2 0xd6d78000107a5c58 in ?? ()<br>Backtrace stopped: previous frame identical to this frame (corrupt stack?)<br><br>I used to see more thatn 20 stacks trace but strangely I see only two. <br>I can still see many stacks for linux-5.4.21 that I was working with in the<br>past. <br>Could anyone check if this happens to anyone?<br>I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build,<br>the kernel will boot ok to the shell prompt.<br>Only the gdb can't show the stack levels.<br><br>My OS : ubuntu-20.04 5.13.0-35-generic<br><br>$ aarch64-none-elf-gdb --version<br>GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11<br>(arm-10.16)) 10.1.90.20201028-git<br>Copyright (C) 2020 Free Software Foundation, Inc.<br>License GPLv3+: GNU GPL version 3 or later<br><<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>This is free software: you are free to change and redistribute it.<br>There is NO WARRANTY, to the extent permitted by law.<br><br>Thank you.<br>Chan Kim<br><br><br><br><br><br>_______________________________________________<br>Kernelnewbies mailing list<br><a href="mailto:Kernelnewbies@kernelnewbies.org" target="_blank">Kernelnewbies@kernelnewbies.org</a><br><a href="https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies" target="_blank">https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies</a><o:p></o:p></span></p></blockquote></div><p class=MsoNormal><span lang=EN-US><br clear=all><br>-- <o:p></o:p></span></p><div><p class=MsoNormal><span lang=EN-US>regards,<br><br>Mulyadi Santosa<br>Freelance Linux trainer and consultant<br><br>blog: <a href="http://the-hydra.blogspot.com" target="_blank">the-hydra.blogspot.com</a><br>training: <a href="http://mulyaditraining.blogspot.com" target="_blank">mulyaditraining.blogspot.com</a><o:p></o:p></span></p></div></div></div></div></div></div></div></body></html>