On Wed, 17 Aug 2011 15:12:22 +0530 amit mehta <gmate.amit@gmail.com> wrote:

On Wed, Aug 17, 2011 at 3:04 PM, amit mehta <gmate.amit@gmail.com> wrote:

...

I've a Linux box which is crashing under a particular test scenario. To debug this issue I was thinking of using the crash (man 8 crash) utility (available by default on recent releases of RHEL and SuSE) . This tool requires the kernel crash dump (vmcore) and the debugging symbols. On my machine, I don't see any crash under /var/crash, Hence please let me know the Here configuration changes which i need to do to enable my machine to dump the kernel image Here

It seems my gmail setup is randomly adding newline character when i draft a mail. Which i'm sure is annoying to those who'll read my mail on this mailing list. Can you please help me resolve this issue.

Read Documentation/email-clients.txt Hope this helps.

-Amit

_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Leonid V. Fedorenchik

 You need to configure kdump  on  your RHEL system to enable it to dump the vmcore file when the panic happens... Kdump configuration is explained below.. There is a crashkernel boot parameter to be passed to kernel to specify the memory reserved for the crash kernel.

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment...

http://www.mindtwist.de/main/linux/11-red-hat/27-how-to-configure-kernel-dum...

http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/lia...

Thanks and Regards, Sumeet

Thank you Sumeet for the link, I've enabled kdump on my machine by following the steps mentioned on the link above(http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment...) But it seems that there is some setup problem on my machine as invoking "crash" is giving me an error. see the logs below. <<<snip from /etc/kdump.cfg>>> #raw /dev/sda5 #ext4 /dev/sda2 #ext4 LABEL=/boot #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937 #net my.server.com:/export/tmp #net user@my.server.com path /var/crash core_collector makedumpfile -c --message-level 1 -d 31 #core_collector cp --sparse=always #link_delay 60 #kdump_post /var/crash/scripts/kdump-post.sh #extra_bins /usr/bin/lftp #disk_timeout 30 #extra_modules gfs2 #options modulename options default halt <<<snip from /etc/kdump.cfg>>> [root@rhel6 boot]# mkinitrd /boot/initrd-2.6.32-71.el6.x86_64kdump.img 2.6.32-71.el6.x86_64 [root@rhel6 ~]# service kdump restart Stopping kdump: [ OK ] Your running kernel is using more than 70% of the amount of[WARNING]u reserved for kdump, you should consider increasing your crashkernel reservation Starting kdump: [ OK ] [root@rhel6 ~]# service kdump status Kdump is operational I hit the crash again and this time there was a vmcore under /var/crash directory [root@rhel6 2011-08-17-17:50]# pwd /var/crash/2011-08-17-17:50 [root@rhel6 2011-08-17-17:50]# du -sh vmcore 43M vmcore These are the two debug rpms that i've installed for kernel symbols: [root@rhel6 ~]# rpm -qa|grep -i debug kernel-debuginfo-common-x86_64-2.6.32-71.7.1.el6.x86_64 kernel-debug-debuginfo-2.6.32-71.7.1.el6.x86_64 my production kernel, from which i've booted: [root@rhel6 ~]# uname -r 2.6.32-71.el6.x86_64 [root@rhel6 ~]# crash -s /usr/lib/debug/lib/modules/2.6.32-71.7.1.el6.x86_64.debug/vmlinux /var/crash/2011-08-17-17\:50/vmcore crash: invalid kernel virtual address: 7180 type: "possible" WARNING: cannot read cpu_possible_map < - ---------------------------- crash: seek error: kernel virtual address: ffffffff8208e980 type: "xtime" -Amit

  It seems there is a bugzilla from Red Hat on this issue(680864) in RHEL 6.0 kernel. Please could you try this issue on RHEL 6.1 kernel ?  Also as mentioned in the https://bugzilla.redhat.com/show_bug.cgi?id=680864, which is similar to your issue, you should be able to see the output of bt command and log command in the crash sesion. Please have a look at and let us know what you see.

Regards, Sumeet

It seems that it was a memory issue(not enough memory) to enable crash dump. Today, after migrating my machine to RHEL 6.1 and bumping up the memory, i was able to get the kernel crash dump and could analyze it using crash utility. Thanks again for your help. <<<snip from sample forced crash>> crash> bt PID: 19208 TASK: ffff8801540200c0 CPU: 3 COMMAND: "bash" #0 [ffff88003778d9e0] machine_kexec at ffffffff810310db #1 [ffff88003778da40] kimage_free_pages at ffffffff810b63b2 #2 [ffff88003778dac8] hvc_write_room at ffffffff8131abc6 #3 [ffff88003778db10] setjmp_pre_handler at ffffffff814dec50 #4 [ffff88003778db40] no_context at ffffffff81040cdb #5 [ffff88003778db90] __bad_area_nosemaphore at ffffffff81040f65 #6 [ffff88003778dbe0] bad_area_access_error at ffffffff8104108e #7 [ffff88003778dc10] __do_page_fault at ffffffff810417b3 #8 [ffff88003778dd30] __unregister_kprobe_top at ffffffff814e0c3e #9 [ffff88003778dd60] oops_end at ffffffff814ddfe5 #10 [ffff88003778dde8] hvc_write_room at ffffffff8131abc6 #11 [ffff88003778de20] destroy_hvc_struct at ffffffff8131ae82 #12 [ffff88003778de70] hvc_hangup at ffffffff8131af3e #13 [ffff88003778dea0] proc_reg_unlocked_ioctl at ffffffff811d4ffe #14 [ffff88003778def0] default_llseek at ffffffff81172718 #15 [ffff88003778df00] destroy_watch at ffffffff810d1b62 #16 [ffff88003778df30] sys_pread64 at ffffffff81173151 RIP: 0000003b158d95e0 RSP: 00007fff4e780008 RFLAGS: 00000202 RAX: 0000000000000001 RBX: ffffffff8100b172 RCX: 00007fff4e780098 RDX: 0000000000000002 RSI: 00007f13cb5d7000 RDI: 0000000000000001 RBP: 00007f13cb5d7000 R8: 000000000000000a R9: 00007f13cb5c8700 R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000000002 R13: 0000003b15b8b780 R14: 0000000000000002 R15: 0000003b15b8b780 ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b <<<snip from sample forced crash>> -Amit