Hi List, I am new to crash / kdump tool and failing some problem as mentioned below. I am referring the Linux Kernel Crash Book (http://www.dedoimedo.com/) and URL http://www.dedoimedo.com/computers/crash.html. I am building modified kernel source (2.6.32 based) and added my modules (for study purpose). Building, installing and booting kernel is successful. I have enabled the options for kdump as mentioned in the book: Enable Kexec system call: CONFIG_KEXEC=y Enable kernel crash dumps: CONFIG_CRASH_DUMP=y Optional: Disable Symmetric Multi-Processing (SMP) support CONFIG_SMP=y Enable sysfs file system support: CONFIG_SYSFS=y Enable /proc/vmcore support: CONFIG_PROC_VMCORE=y Configure the kernel with debug info: CONFIG_DEBUG_INFO=y Configure the start section for reserved RAM for the crash kernel: CONFIG_PHYSICAL_START=0x200000 (2MB) Configure kdump kernel so it can be identified: CONFIG_LOCALVERSION="-crash" Kdump configuration /etc/sysconfig/kdump: KDUMP_KERNELVER="" KDUMP_COMMANDLINE="" KDUMP_COMMANDLINE_APPEND="maxcpus=1 " KEXEC_OPTIONS="" KDUMP_IMMEDIATE_REBOOT="yes" KDUMP_TRANSFER="" KDUMP_SAVEDIR="file:///var/crash" KDUMP_KEEP_OLD_DUMPS="5" KDUMP_FREE_DISK_SIZE="64" KDUMP_VERBOSE="3" KDUMP_DUMPLEVEL="0" KDUMP_DUMPFORMAT="compressed" There is no option KDUMP_DUMPDEV option There is no option KDUMP_RUNLEVEL I booted successfully with this kernel and tried to crash it by module. After rebooting, I found that vmcore is generated under /var/crash/ But I am not able to analyze it with crash command. linux:/home/adil # cat /proc/cmdline root=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part1 resume=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part2 splash=silent crashkernel=256M-:128M vga=0x31a linux:/home/adil # linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash /boot/vmlinuz-2.6.32.12-crash-crash crash 5.0.1 Copyright (C) 2002-2010 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format Usage: crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile] Enter "crash -h" for details. linux:/home/adil # The URL http://www.dedoimedo.com/computers/crash.html mentioned that "The newer versions of Kdump can work with compressed kernel images. Furthermore, they copy the System map file and the kernel image into the crash directory, making the use of crash utility somewhat simpler." linux:/home/adil # ls -al /var/crash/2012-01-30-18\:08/ total 1335536 drwxr-xr-x 2 root root 4096 2012-01-30 18:13 . drwxr-xr-x 8 root root 4096 2012-01-31 12:17 .. -rw-r--r-- 1 root root 187 2012-01-30 18:13 README.txt -rw-r--r-- 1 root root 1716605 2012-01-30 18:13 System.map-2.6.32.12-0.7-default -rw------- 1 root root 1360732590 2012-01-30 18:13 vmcore -rw-r--r-- 1 root root 3774506 2012-01-30 18:13 vmlinux-2.6.32.12-0.7-default.gz linux:/home/adil # ls -al /var/crash/2012-01-31-12\:17/ total 1343860 drwxr-xr-x 2 root root 4096 2012-01-31 12:24 . drwxr-xr-x 8 root root 4096 2012-01-31 12:17 .. -rw-r--r-- 1 root root 187 2012-01-31 12:24 README.txt -rw------- 1 root root 1374748735 2012-01-31 12:24 vmcore linux:/home/adil # linux:/home/adil # ls /boot/ backup_mbr boot boot.readme config-2.6.32.12-0.7-default config-2.6.32.12-0.7-xen grub initrd initrd-2.6.32.12-0.7-default initrd-2.6.32.12-0.7-default-kdump initrd-2.6.32.12-0.7-xen initrd-2.6.32.12-crash-crash initrd-2.6.32.12-crash-crash-kdump initrd-xen message symsets-2.6.32.12-0.7-default.tar.gz symtypes-2.6.32.12-0.7-default.gz symvers-2.6.32.12-0.7-default.gz symvers-2.6.32.12-0.7-xen.gz System.map-2.6.32.12-0.7-default System.map-2.6.32.12-0.7-xen System.map-2.6.32.12-crash-crash vmlinux-2.6.32.12-0.7-xen.gz vmlinuz vmlinuz-2.6.32.12-0.7-default vmlinuz-2.6.32.12-0.7-xen vmlinuz-2.6.32.12-crash-crash vmlinuz-xen vmlinux-2.6.32.12-0.7-default.gz linux:/home/adil # Another observation is boot.kdump seems to on but manually start giving me error: linux:/home/adil # chkconfig boot.kdump boot.kdump on linux:/home/adil # linux:/home/adil # /etc/init.d/boot.kdump start Loading kdump Regenerating kdump initrd ... Can't find kernel text map area from kcore Cannot load /boot/vmlinuz-2.6.32.12-crash-crash failed linux:/home/adil # Other Query: Following is not clear mentioned in the book under "section 11.2 Crash (capture) kernel": -------------- This means that while your production kernels will most likely be named vmlinuz, the Kdump crash kernels need to be uncompressed, hence named vmlinux, or rather vmlinux-kdump. --------------- Please help how to correctly setup and use crash on my machine. Thank you, Adil
Hi Adil... On Mon, Feb 6, 2012 at 18:03, Adil Mujeeb <mujeeb.adil@gmail.com> wrote:
crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format
I am not kdump/crash user at all, but the above error message seems to give a hint that you indeed need to use vmlinux (uncompressed image) instead of vmlinuz (compressed image). Since you said you compile the kernel image by yourself, I am sure you have it, right? -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash /boot/vmlinuz-2.6.32.12-crash-crash
crash 5.0.1 Copyright (C) 2002-2010 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details.
crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format
Usage: crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
Adil, If you have a RHEL 6.1 then kdump installation and setup is a piece of cake, Luckily I have RHEL 6.1 on my machine and had setup kdump by following the instructions in the link below: http://people.redhat.com/anderson/crash_whitepaper/ To use crash utility for analyzing kernel crashes here's a sample invocation of this command on my machine. [root@rhel 127.0.0.1-2011-11-24-12:57:47]# cat /etc/issue Red Hat Enterprise Linux Server release 6.1 (Santiago) Kernel \r on an \m [root@rhel 127.0.0.1-2011-11-24-12:57:47]# rpm -qa|grep -i debug kernel-debuginfo-2.6.32-131.0.15.el6.x86_64 kernel-debuginfo-common-x86_64-2.6.32-131.0.15.el6.x86_64 [root@rhel 127.0.0.1-2011-11-24-12:57:47]# pwd /var/crash/127.0.0.1-2011-11-24-12:57:47 [root@rhel 127.0.0.1-2011-11-24-12:57:47]# ls vmcore [root@lancer-evt03 127.0.0.1-2011-11-24-12:57:47]# crash -s /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux vmcore crash: page excluded: kernel virtual address: ffffffffffffffff type: "possible" WARNING: cannot read cpu_possible_map crash: page excluded: kernel virtual address: ffffffffffffffff type: "present" WARNING: cannot read cpu_present_map crash: page excluded: kernel virtual address: ffffffffffffffff type: "online" WARNING: cannot read cpu_online_map WARNING: kernels compiled by different gcc versions: /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux: 4.4.5 vmcore kernel: 4.4.4 crash: page excluded: kernel virtual address: ffffffffffffffff type: "cpu_present_map" crash: page excluded: kernel virtual address: ffffffffffffffff type: "cpu_present_map" crash> log Pid: 10679, comm: bash Tainted: P W ---------------- T 2.6.32-131.0.15.el6.x86_64 #1 S5000PAL RIP: 0010:[<ffffffff8115a1d4>] [<ffffffff8115a1d4>] cache_alloc_refill+0x1e4/0x240 RSP: 0018:ffff880157997bd8 EFLAGS: 00010046 RAX: 0000000000000009 RBX: ffff88015fc10080 RCX: 000000000000001e RDX: ffff8801586ef000 RSI: ffff88015fc224c0 RDI: ffff880157fa9000 RBP: ffff880157997c38 R08: ffff8801586ef000 R09: 0000000070616d5f R10: 0000000013131313 R11: 0000000000000000 R12: ffff88015ac54400 R13: ffff88015fc224c0 R14: 0000000000000009 R15: ffff880157fa9000 FS: 00007f31a13ab700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000001f084a8 CR3: 00000001559d5000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process bash (pid: 10679, threadinfo ffff880157996000, task ffff8801575c0a80) Stack: ffff880158ef2cc0 0000000058ef2cc0 ffff88015fc22500 000492d0a028816e <0> ffff88015fc224e0 ffff88015fc224d0 0000000000000020 0000000000000044 <0> 0000000000000010 00000000000080d0 ffff88015fc10080 ffffffffa027fd07 Call Trace: [<ffffffffa027fd07>] ? ext4_htree_store_dirent+0x37/0x140 [ext4] [<ffffffff8115b079>] __kmalloc+0x1a9/0x220 [<ffffffffa027fd07>] ext4_htree_store_dirent+0x37/0x140 [ext4] [<ffffffffa028d338>] htree_dirblock_to_tree+0x128/0x190 [ext4] [<ffffffffa028ddaa>] ext4_htree_fill_tree+0x16a/0x260 [ext4] [<ffffffffa027fb75>] ext4_readdir+0x4f5/0x650 [ext4] [<ffffffff81185f90>] ? filldir+0x0/0xe0 [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150 [<ffffffff81185f90>] ? filldir+0x0/0xe0 [<ffffffff81186210>] vfs_readdir+0xc0/0xe0 [<ffffffff81186399>] sys_getdents+0x89/0xf0 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b Code: 89 ff e8 a0 8c 11 00 eb 99 66 0f 1f 44 00 00 41 c7 45 60 01 00 00 00 4d 8b 7d 20 4c 39 7d c0 0f 85 f2 fe ff ff eb 84 0f 0b eb fe <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 eb f4 8b 55 ac 8b 75 bc 31 RIP [<ffffffff8115a1d4>] cache_alloc_refill+0x1e4/0x240 RSP <ffff880157997bd8> crash> This crash was invoked by a module to delibrately cause kernel crash. -Amit
participants (3)
-
Adil Mujeeb -
Kumar Mehta -
Mulyadi Santosa