Help needed in getting kernel dump in QEMU VM

[Baoquan He]

On 11/15/21 at 03:49pm, Dongliang Mu wrote:
> On Mon, Nov 15, 2021 at 3:27 PM Baoquan He <bhe at redhat.com> wrote:
> >
> > Hi,
> >
> > On 11/13/21 at 10:40am, Dongliang Mu wrote:
> > > Hi all KDUMP maintainers,
> > >
> > > I would like to generate a kernel dump within QEMU VM.
> > >
> > > 1. I reproduced the kernel crash [1] in QEMU VM. The QEMU startup
> > > script is as follows:
> > >
> > > qemu-system-x86_64 \
> > >   -kernel $KERNEL/arch/x86/boot/bzImage \
> > >   -append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
> > >   -hda $IMAGE/stretch.img \
> > >   -net user,hostfwd=tcp::10021-:22 -net nic \
> > >   -enable-kvm \
> > >   -nographic \
> > >   -m 2G \
> > >   -smp 2 \
> > >   -pidfile vm.pid \
> > >   2>&1 | tee vm.log
> > >
> > > The stretch.img is generated by Syzkaller script [1]. -kernel option
> > > is convenient for loading any other kernels.
> > >
> > > 2. As the .config already has the essential
> > > configuration(CONFIG_KEXEC, CONFIG_CRASH_DUMP, CONFIG_DEBUG_INFO), I
> > > did not change this configuration file.
> > >
> > > 3. I installed kdump-tools crash kexec-tools makedumpfile
> > > linux-image-4.9.0-13-amd64 in the stretch.img. Here I installed
> > > linux-image-4.9.0-13-amd64 because there is no default kernel in /boot
> > > directory. And to make kdump-tools working, I modify
> > > /etc/default/kdump-tools in the following:
> > >
> > > KDUMP_INITRD=/boot/initrd.img-4.9.0-13-amd64
> > > KDUMP_KERNEL=/boot/vmlinuz-4.9.0-13-amd64
> >
> > What distros are you using? Asking this because I am sure you are not
> > using Fedora/RHEL OS. The implementation of kdump tools is different in
> > each distros, even though the mechanims in kdump code is the same.
> >
> 
> I am using Debian stretch as the guest OS. So kdump-tools kexec
> makedumpfile is all from Debian.

Then I would sugest asking in a Debian/ubuntu forum or mailing list, figure
out if the configuration or setting is correct. 

I never try Debian OS, can't help, sorry.

> 
> > When we try to get help from upstream, considering and asking good question
> > is very important for getting quick response and effective help.
> >
> > Thanks
> > Baoquan
> >
> > >
> > > 4. I append "crashkernel=384M-:128M" to the command line in the
> > > startup script of QEMU.
> > >
> > > 5. After rebooting, kdump service can start successfully, and the
> > > kdump-config shows:
> > >
> > > root at syzkaller:~# kdump-config show
> > > DUMP_MODE:        kdump
> > > USE_KDUMP:        1
> > > KDUMP_SYSCTL:     kernel.panic_on_oops=1
> > > KDUMP_COREDIR:    /var/crash
> > > crashkernel addr: 0x77000000
> > >    /boot/vmlinuz-4.9.0-13-amd64
> > > kdump initrd:
> > >    /boot/initrd.img-4.9.0-13-amd64
> > > current state:    ready to kdump
> > >
> > > kexec command:
> > >   /sbin/kexec -p --command-line="earlyprintk=serial oops=panic
> > > panic_on_warn=1 nmi_watchdog=panic panic=86400 net.ifnames=0
> > > sysctl.kernel.hung_task_all_cpu_backtrace=1 ima_policy=tcb
> > > kvm-intel.nested=1 nf-conntrack-ftp.ports=20000
> > > nf-conntrack-tftp.ports=20000 nf-conntrack-sip.ports=20000
> > > nf-conntrack-irc.ports=20000 nf-conntrack-sane.ports=20000
> > > vivid.n_devs=16 vivid.multiplanar=1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2
> > > netrom.nr_ndevs=16 rose.rose_ndevs=16 spec_store_bypass_disable=prctl
> > > numa=fake=2 nopcid dummy_hcd.num=8 binder.debug_mask=0
> > > rcupdate.rcu_expedited=1 root=/dev/sda console=ttyS0 vsyscall=native
> > > watchdog_thresh=55 workqueue.watchdog_thresh=140 console=ttyS0
> > > root=/dev/sda debug earlyprintk=serial slub_debug=QUZ irqpoll
> > > nr_cpus=1 nousb systemd.unit=kdump-tools.service
> > > ata_piix.prefer_ms_hyperv=0" --initrd=/boot/initrd.img-4.9.0-13-amd64
> > > /boot/vmlinuz-4.9.0-13-amd64
> > >
> > > 6. When I execute the PoC, the current kernel crashes and then reboots
> > > into the dump-capture kernel. However, the kernel log shows, it is in
> > > emergency mode,
> > >
> > > You are in emergency mode. After logging in, type "journalctl -xb" to view
> > > system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
> > > try again to boot into default mode.
> > >
> > > Finally, I would like to ask several questions:
> > > 1) is the emergency mode due to the incorrect command line?
> > > 2) is this the right way to generate kernel dump from QEMU VM?
> > > 3) Any comments on the above procedures?
> > >
> > > Thanks very much in advance.
> > >
> > > [1] general protection fault in reiserfs_security_init
> > > (https://syzkaller.appspot.com/bug?id=8abaedbdeb32c861dc5340544284167dd0e46cde)
> > >
> > > [2] https://github.com/google/syzkaller/blob/master/tools/create-image.sh
> > >
> > > --
> > > My best regards to you.
> > >
> > >      No System Is Safe!
> > >      Dongliang Mu
> >
> >
>