kernel crash dump analysis using crash utility
My Linux box just crashed while performing some network related tests. I'm trying to analyze the kernel crash dump using "Crash" utility. Need your help in analyzing it. <<<Snip from crash output>>> # crash /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux vmcore .........versioning information here............... .................etc etc.......................... KERNEL: /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux DUMPFILE: vmcore [PARTIAL DUMP] CPUS: 32 DATE: Mon Sep 19 09:44:45 2011 <---- --- UPTIME: 2 days, 22:55:44 LOAD AVERAGE: 0.00, 0.02, 0.00 TASKS: 409 NODENAME: RHEL61ga RELEASE: 2.6.32-131.0.15.el6.x86_64 VERSION: #1 SMP Tue May 10 15:42:40 EDT 2011 MACHINE: x86_64 (2660 Mhz) MEMORY: 12 GB PANIC: "Oops: 0000 [#1] SMP " (check log for details) PID: 0 COMMAND: "swapper" TASK: ffffffff81a2d020 (1 of 8) [THREAD_INFO: ffffffff81a00000] CPU: 0 STATE: TASK_RUNNING (ACTIVE) WARNING: panic task not found ........................ stack trace frame 0 - frame 18 here..................... ............................................................................................... --- <IRQ stack> --- #19 [ffffffff81a01da8] ret_from_intr at ffffffff8100bad3 [exception RIP: acpi_check_resource_conflict+207] < ---- --- -------- RIP: ffffffff812bb86e RSP: ffffffff81a01e58 RFLAGS: 00000206 RAX: 0000000000000000 RBX: ffffffff81a01ec8 RCX: 0000000000000000 RDX: 0000000000000006 RSI: 0000000000000000 RDI: 00000000000018f6 RBP: ffffffff8100bace R8: 0000000000000000 R9: 0000000000000ca3 R10: ffff88019aa061c2 R11: ffff88019aa06201 R12: ffffffff81b7c0b8 R13: 0000000000000001 R14: ffffffff810ece03 R15: ffffffff81a01dd8 ORIG_RAX: ffffffffffffffb5 CS: 0010 SS: 0018 #20 [ffffffff81a01e50] acpi_check_resource_conflict at ffffffff812bb851 #21 [ffffffff81a01ed0] show_current_driver at ffffffff813eccb7 #22 [ffffffff81a01ef0] cpu_idle at ffffffff81009e96 <<<Snip from crash output>>> Observation: i) The panic hit at Sep 19 09:44:45 2011, and a peek inside /var/log/messages reveals that the machine was down for about 4-5 minutes around this timestamp. But there are no other relevant information available in the log file, so as printed in the panic string above: "Oops: 0000 [#1] SMP " (check log for details)", where else should i look for logs ? ii) Does frame 19:: [exception RIP: acpi_check_resource_conflict+207] and the immediated dump of the registers of my machine(Mine is x86_64) is an indication of where the Oops occured ? iii) Disassembly of "acpi_check_resource_conflict" at an offset:207 shows a 'test' instruction being carried out on CPU 0: 0xffffffff812bb86e <acpi_check_resource_conflict+207>: test %ebx,0xc4c3e8(%rip) # 0xffffffff81f07c5c iv) I belive that on 64 bit x86 machines, RIP represents EIP and similarly RBX represents EBX register. hence at the time of panic, the contents fo these registers were as follows: RBX = ffffffff81a01ec8 RIP = ffffffff812bb86e I don't know much of assembly, just did little bit of search on web, and my understanding is that the instruction "test %ebx,0xc4c3e8(%rip)" tells the processor to do the bitwise "and" of contents of ebx register with the value at the offset:0xc4c3e8 from the base address pointed by rip. i.e. ffffffff812bb86e+c4c3e8 = FFFFFFFF81F07C56 ? Please correct me if I'm wrong. -Amit
On Mon, Sep 19, 2011 at 12:26 PM, amit mehta <gmate.amit@gmail.com> wrote:
My Linux box just crashed while performing some network related tests. I'm trying to analyze the kernel crash dump using "Crash" utility. Need your help in analyzing it.
<<<Snip from crash output>>> # crash /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux vmcore .........versioning information here............... .................etc etc.......................... KERNEL: /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux DUMPFILE: vmcore [PARTIAL DUMP] CPUS: 32 DATE: Mon Sep 19 09:44:45 2011 <---- --- UPTIME: 2 days, 22:55:44 LOAD AVERAGE: 0.00, 0.02, 0.00 TASKS: 409 NODENAME: RHEL61ga RELEASE: 2.6.32-131.0.15.el6.x86_64 VERSION: #1 SMP Tue May 10 15:42:40 EDT 2011 MACHINE: x86_64 (2660 Mhz) MEMORY: 12 GB PANIC: "Oops: 0000 [#1] SMP " (check log for details) PID: 0 COMMAND: "swapper" TASK: ffffffff81a2d020 (1 of 8) [THREAD_INFO: ffffffff81a00000] CPU: 0 STATE: TASK_RUNNING (ACTIVE) WARNING: panic task not found
........................ stack trace frame 0 - frame 18 here..................... ...............................................................................................
--- <IRQ stack> --- #19 [ffffffff81a01da8] ret_from_intr at ffffffff8100bad3 [exception RIP: acpi_check_resource_conflict+207] < ---- --- -------- RIP: ffffffff812bb86e RSP: ffffffff81a01e58 RFLAGS: 00000206 RAX: 0000000000000000 RBX: ffffffff81a01ec8 RCX: 0000000000000000 RDX: 0000000000000006 RSI: 0000000000000000 RDI: 00000000000018f6 RBP: ffffffff8100bace R8: 0000000000000000 R9: 0000000000000ca3 R10: ffff88019aa061c2 R11: ffff88019aa06201 R12: ffffffff81b7c0b8 R13: 0000000000000001 R14: ffffffff810ece03 R15: ffffffff81a01dd8 ORIG_RAX: ffffffffffffffb5 CS: 0010 SS: 0018 #20 [ffffffff81a01e50] acpi_check_resource_conflict at ffffffff812bb851 #21 [ffffffff81a01ed0] show_current_driver at ffffffff813eccb7 #22 [ffffffff81a01ef0] cpu_idle at ffffffff81009e96
<<<Snip from crash output>>> This same utility(crash) has an option of dumping system message buffer by invoking "log". For some reason, kernel message buffer did not got flushed on the disk and hence there were no relevant information on disk file. i.e. /var/log/messages. Using this same feature(log) i could find the perpetrator :)
-Amit
On Mon, Sep 19, 2011 at 13:56, amit mehta <gmate.amit@gmail.com> wrote:
...............................................................................................
--- <IRQ stack> --- #19 [ffffffff81a01da8] ret_from_intr at ffffffff8100bad3 [exception RIP: acpi_check_resource_conflict+207] < ---- --- --------
could you post the complete log somewhere? BTW, looking at http://lxr.linux.no/#linux+v2.6.32.46/drivers/acpi/osl.c#L1177, looks like it is a procedure to find out potential conflict in ACPI..and maybe, in your case, it hits hard. -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
participants (2)
-
amit mehta -
Mulyadi Santosa