kernel crash dump analysis using crash utility

[amit mehta]

On Mon, Sep 19, 2011 at 12:26 PM, amit mehta <gmate.amit at gmail.com> wrote:
> My Linux box just crashed while performing some network related tests.
> I'm trying to analyze the kernel crash dump using "Crash" utility.
> Need your help in analyzing it.
>
> <<<Snip from crash output>>>
> # crash /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux vmcore
> .........versioning information here...............
> .................etc etc..........................
>      KERNEL: /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux
>    DUMPFILE: vmcore  [PARTIAL DUMP]
>        CPUS: 32
>        DATE: Mon Sep 19 09:44:45 2011   <---- ---
>      UPTIME: 2 days, 22:55:44
> LOAD AVERAGE: 0.00, 0.02, 0.00
>       TASKS: 409
>    NODENAME: RHEL61ga
>     RELEASE: 2.6.32-131.0.15.el6.x86_64
>     VERSION: #1 SMP Tue May 10 15:42:40 EDT 2011
>     MACHINE: x86_64  (2660 Mhz)
>      MEMORY: 12 GB
>       PANIC: "Oops: 0000 [#1] SMP " (check log for details)
>         PID: 0
>     COMMAND: "swapper"
>        TASK: ffffffff81a2d020  (1 of 8)  [THREAD_INFO: ffffffff81a00000]
>         CPU: 0
>       STATE: TASK_RUNNING (ACTIVE)
>     WARNING: panic task not found
>
> ........................ stack trace frame 0 - frame 18
> here.....................
> ...............................................................................................
>
> --- <IRQ stack> ---
> #19 [ffffffff81a01da8] ret_from_intr at ffffffff8100bad3
>    [exception RIP: acpi_check_resource_conflict+207] < ---- --- --------
>    RIP: ffffffff812bb86e  RSP: ffffffff81a01e58  RFLAGS: 00000206
>    RAX: 0000000000000000  RBX: ffffffff81a01ec8  RCX: 0000000000000000
>    RDX: 0000000000000006  RSI: 0000000000000000  RDI: 00000000000018f6
>    RBP: ffffffff8100bace   R8: 0000000000000000   R9: 0000000000000ca3
>    R10: ffff88019aa061c2  R11: ffff88019aa06201  R12: ffffffff81b7c0b8
>    R13: 0000000000000001  R14: ffffffff810ece03  R15: ffffffff81a01dd8
>    ORIG_RAX: ffffffffffffffb5  CS: 0010  SS: 0018
> #20 [ffffffff81a01e50] acpi_check_resource_conflict at ffffffff812bb851
> #21 [ffffffff81a01ed0] show_current_driver at ffffffff813eccb7
> #22 [ffffffff81a01ef0] cpu_idle at ffffffff81009e96
>
> <<<Snip from crash output>>>
This same utility(crash) has an option of dumping system message buffer
by invoking "log". For some reason, kernel message buffer did not got flushed
on the disk and hence there were no relevant information on disk file. i.e.
/var/log/messages.
Using this same feature(log) i could find the perpetrator :)

-Amit