Help needed in getting kernel dump in QEMU VM

[Dongliang Mu]

On Mon, Nov 15, 2021 at 3:27 PM Baoquan He <bhe at redhat.com> wrote:
>
> Hi,
>
> On 11/13/21 at 10:40am, Dongliang Mu wrote:
> > Hi all KDUMP maintainers,
> >
> > I would like to generate a kernel dump within QEMU VM.
> >
> > 1. I reproduced the kernel crash [1] in QEMU VM. The QEMU startup
> > script is as follows:
> >
> > qemu-system-x86_64 \
> >   -kernel $KERNEL/arch/x86/boot/bzImage \
> >   -append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
> >   -hda $IMAGE/stretch.img \
> >   -net user,hostfwd=tcp::10021-:22 -net nic \
> >   -enable-kvm \
> >   -nographic \
> >   -m 2G \
> >   -smp 2 \
> >   -pidfile vm.pid \
> >   2>&1 | tee vm.log
> >
> > The stretch.img is generated by Syzkaller script [1]. -kernel option
> > is convenient for loading any other kernels.
> >
> > 2. As the .config already has the essential
> > configuration(CONFIG_KEXEC, CONFIG_CRASH_DUMP, CONFIG_DEBUG_INFO), I
> > did not change this configuration file.
> >
> > 3. I installed kdump-tools crash kexec-tools makedumpfile
> > linux-image-4.9.0-13-amd64 in the stretch.img. Here I installed
> > linux-image-4.9.0-13-amd64 because there is no default kernel in /boot
> > directory. And to make kdump-tools working, I modify
> > /etc/default/kdump-tools in the following:
> >
> > KDUMP_INITRD=/boot/initrd.img-4.9.0-13-amd64
> > KDUMP_KERNEL=/boot/vmlinuz-4.9.0-13-amd64
>
> What distros are you using? Asking this because I am sure you are not
> using Fedora/RHEL OS. The implementation of kdump tools is different in
> each distros, even though the mechanims in kdump code is the same.
>

I am using Debian stretch as the guest OS. So kdump-tools kexec
makedumpfile is all from Debian.

> When we try to get help from upstream, considering and asking good question
> is very important for getting quick response and effective help.
>
> Thanks
> Baoquan
>
> >
> > 4. I append "crashkernel=384M-:128M" to the command line in the
> > startup script of QEMU.
> >
> > 5. After rebooting, kdump service can start successfully, and the
> > kdump-config shows:
> >
> > root at syzkaller:~# kdump-config show
> > DUMP_MODE:        kdump
> > USE_KDUMP:        1
> > KDUMP_SYSCTL:     kernel.panic_on_oops=1
> > KDUMP_COREDIR:    /var/crash
> > crashkernel addr: 0x77000000
> >    /boot/vmlinuz-4.9.0-13-amd64
> > kdump initrd:
> >    /boot/initrd.img-4.9.0-13-amd64
> > current state:    ready to kdump
> >
> > kexec command:
> >   /sbin/kexec -p --command-line="earlyprintk=serial oops=panic
> > panic_on_warn=1 nmi_watchdog=panic panic=86400 net.ifnames=0
> > sysctl.kernel.hung_task_all_cpu_backtrace=1 ima_policy=tcb
> > kvm-intel.nested=1 nf-conntrack-ftp.ports=20000
> > nf-conntrack-tftp.ports=20000 nf-conntrack-sip.ports=20000
> > nf-conntrack-irc.ports=20000 nf-conntrack-sane.ports=20000
> > vivid.n_devs=16 vivid.multiplanar=1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2
> > netrom.nr_ndevs=16 rose.rose_ndevs=16 spec_store_bypass_disable=prctl
> > numa=fake=2 nopcid dummy_hcd.num=8 binder.debug_mask=0
> > rcupdate.rcu_expedited=1 root=/dev/sda console=ttyS0 vsyscall=native
> > watchdog_thresh=55 workqueue.watchdog_thresh=140 console=ttyS0
> > root=/dev/sda debug earlyprintk=serial slub_debug=QUZ irqpoll
> > nr_cpus=1 nousb systemd.unit=kdump-tools.service
> > ata_piix.prefer_ms_hyperv=0" --initrd=/boot/initrd.img-4.9.0-13-amd64
> > /boot/vmlinuz-4.9.0-13-amd64
> >
> > 6. When I execute the PoC, the current kernel crashes and then reboots
> > into the dump-capture kernel. However, the kernel log shows, it is in
> > emergency mode,
> >
> > You are in emergency mode. After logging in, type "journalctl -xb" to view
> > system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
> > try again to boot into default mode.
> >
> > Finally, I would like to ask several questions:
> > 1) is the emergency mode due to the incorrect command line?
> > 2) is this the right way to generate kernel dump from QEMU VM?
> > 3) Any comments on the above procedures?
> >
> > Thanks very much in advance.
> >
> > [1] general protection fault in reiserfs_security_init
> > (https://syzkaller.appspot.com/bug?id=8abaedbdeb32c861dc5340544284167dd0e46cde)
> >
> > [2] https://github.com/google/syzkaller/blob/master/tools/create-image.sh
> >
> > --
> > My best regards to you.
> >
> >      No System Is Safe!
> >      Dongliang Mu
>
>