kernel 5.5 kernel and tg3 driver
Hello Experts, I was using click-1.6.0 on centos 5.3 kernel and recently* ported that to centos5.5 kernel.*. After porting, am getting the (*low stack detected by irq handler) and the *following errors on the console frequently and causing the system unresponsive totally. Even after reboot, this issue is re-occurring some time after every 3 days. System info: kernel: centos 5.5 (2.6.194-el5) kernel Modules using: click router NIC driver: tg3 (broadcom) Note: The above setup was working successfully for years earlier. I suspect tg3 driver is doing something here but not sure of how to confirm this. Am looking for pointers to start with so that I can investigate. I tried to do web search on this but not useful. Also, would like to know any ways/tools to debug using the addresses given in the stack trace. Thanks in advance. ######################## *START OF LOGS* ############################ Oct 1 14:16:56 server03 kernel: low stack detected by irq handler Oct 1 14:16:56 server03 kernel: [<c0405e9b>] do_IRQ+0x86/0xc3 Oct 1 14:16:56 server03 kernel: [<c04046e6>] common_interrupt+0x1a/0x20 Oct 1 14:16:56 server03 kernel: [<c05a5c06>] neigh_lookup+0x23/0x7f Oct 1 14:16:56 server03 kernel: [<c05d6800>] arp_bind_neighbour+0x50/0x71 Oct 1 14:16:56 server03 kernel: [<c05b7b44>] rt_intern_hash+0x372/0x480 Oct 1 14:16:56 server03 kernel: [<c05b8352>] __ip_route_output_key+0x700/ 0x79e Oct 1 14:16:56 server03 kernel: [<c05b83d4>] __ip_route_output_key+0x782/0x79e Oct 1 14:16:57 server03 kernel: [<c05b8403>] ip_route_output_flow+0x13/0x1d4 Oct 1 14:16:57 server03 kernel: [<c05b85cd>] ip_route_output_key+0x9/0xb Oct 1 14:16:57 server03 kernel: [<c05bdd5d>] ip_send_reply+0xf8/0x209 Oct 1 14:16:57 server03 kernel: [<c045f1a6>] cache_alloc_refill+0x122/0x3e4 Oct 1 14:16:57 server03 kernel: [<c05b7c2f>] rt_intern_hash+0x45d/0x480 Oct 1 14:16:57 server03 kernel: [<c05cfc56>] tcp_v4_send_reset+0x104/0x11c Oct 1 14:16:57 server03 kernel: [<c05d0879>] tcp_v4_do_rcv+0x240/0x269 Oct 1 14:16:57 server03 kernel: [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848 Oct 1 14:16:57 server03 kernel: [<c05bb173>] ip_local_deliver+0x157/0x1d0 Oct 1 14:16:57 server03 kernel: [<c05baff2>] ip_rcv+0x3cb/0x3f5 Oct 1 14:16:57 server03 kernel: [<c05a141d>] __netif_receive_skb+0x3b3/0x3d5 Oct 1 14:16:57 server03 kernel: [<fb1b121e>] _ZN6ToHost4pushEiP6Packet+0xee/0x210 [click] Oct 1 14:16:57 server03 kernel: [<fb1ce85d>] _ZN12EnsureEther24pushEiP6Packet+0x1d/0x20 [click] Oct 1 14:16:57 server03 kernel: [<fb1c0895>] _ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click] Oct 1 14:16:57 server03 kernel: [<c05bb173>] ip_local_deliver+0x157/0x1d0 Oct 1 14:16:57 server03 kernel: [<c05baff2>] ip_rcv+0x3cb/0x3f5 Oct 1 14:16:57 server03 kernel: [<fb19e42d>] _ZN4IPRw7Mapping9make_pairEiRK8IPFlowIDS3_iiPS0_S4_+0xad/0xd0 [click] Oct 1 14:16:57 server03 kernel: [<fb19fcfb>] _ZN4IPRw7Pattern14create_mappingEiRK8IPFlowIDiiPNS_7MappingES5_RK7HashMapIS1_S5_E+0x1cb/0x1f0 [click] Oct 1 14:16:57 server03 kernel: [<fb218ce3>] _ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160 [click] Oct 1 14:16:57 server03 kernel: [<fb218d9e>] _ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click] Oct 1 14:16:57 server03 kernel: [<fb1ea535>] _ZN11MacThrottle10l2PushProcEiP6Packet+0x55/0x3b0 [click] Oct 1 14:16:57 server03 kernel: [<fb1c0895>] _ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click] Oct 1 14:16:57 server03 kernel: [<fb218ce3>] _ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160 [click] Oct 1 14:16:57 server03 kernel: [<fb218d9e>] _ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click] Oct 1 14:16:58 server03 kernel: [<fb1eab96>] _ZN11MacThrottle4pushEiP6Packet+0x306/0x1890 [click] Oct 1 14:16:58 server03 kernel: [<c0417f22>] enqueue_task+0x29/0x39 Oct 1 14:16:58 server03 kernel: [<c041805a>] __activate_task+0x4a/0x59 Oct 1 14:16:58 server03 kernel: [<fb1c69d8>] _ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click] Oct 1 14:16:58 server03 kernel: [<fb155806>] _ZN7Element4pushEiP6Packet+0x26/0x40 [click] Oct 1 14:16:58 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:58 server03 kernel: [<fb1ee075>] _ZN17NtlmLoginDetector4pushEiP6Packet+0x65/0x15d0 [click] Oct 1 14:16:58 server03 kernel: [<fb155806>] _ZN7Element4pushEiP6Packet+0x26/0x40 [click] Oct 1 14:16:58 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:58 server03 kernel: [<fb21f881>] _ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click] Oct 1 14:16:58 server03 kernel: [<fb1fbfdb>] _ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click] Oct 1 14:16:58 server03 kernel: [<c05bb173>] ip_local_deliver+0x157/0x1d0 Oct 1 14:16:58 server03 kernel: [<fb1d8e86>] _ZN11HashIPTable4pushEiP6Packet+0xf6/0x2d0 [click] Oct 1 14:16:58 server03 kernel: [<c05a141d>] __netif_receive_skb+0x3b3/0x3d5 Oct 1 14:16:58 server03 kernel: [<fb1c69d8>] _ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click] Oct 1 14:16:58 server03 kernel: [<fb192904>] _ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click] Oct 1 14:16:58 server03 kernel: [<fb1e4517>] _ZN12IPPortLookup4pushEiP6Packet+0x47/0x9e0 [click] Oct 1 14:16:58 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f Oct 1 14:16:58 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3 Oct 1 14:16:58 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f Oct 1 14:16:58 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:58 server03 kernel: [<fb1cd5fc>] _ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click] Oct 1 14:16:59 server03 kernel: [<fb1c69d8>] _ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click] Oct 1 14:16:59 server03 kernel: [<fb1c69d8>] _ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1cdbb8>] _ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click] Oct 1 14:16:59 server03 kernel: [<fb1b1853>] _ZN11ARPQuerier214send_query_forERK9IPAddressitit+0x133/0x3f0 [click] Oct 1 14:16:59 server03 kernel: [<fb1b1dba>] _ZN11ARPQuerier29handle_ipEP6Packet+0x2aa/0x5f0 [click] Oct 1 14:16:59 server03 kernel: [<fb1b31bd>] _ZN11ARPQuerier24pushEiP6Packet+0xcd/0x110 [click] Oct 1 14:16:59 server03 kernel: [<fb1cdbb8>] _ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1cdbb8>] _ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1e234f>] _ZN13IPFilterGroup4pushEiP6Packet+0x19f/0xa80 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:16:59 server03 kernel: [<c05a5c59>] neigh_lookup+0x76/0x7f Oct 1 14:16:59 server03 kernel: [<fb1d3492>] _ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click] Oct 1 14:16:59 server03 kernel: [<fb1e66cf>] _ZN15LinearIPFilter24pushEiP6Packet+0x4f/0x3f0 [click] Oct 1 14:16:59 server03 kernel: [<fb1b80e8>] _ZN11ARPQuerier420handle_arp_retaggingEP6Packett+0x58/0x940 [click] Oct 1 14:16:59 server03 kernel: [<c05bf82d>] ip_output+0x1f0/0x227 Oct 1 14:16:59 server03 kernel: [<fb1dd289>] _ZN14HashMACLookup54pushEiP6Packet+0x249/0x890 [click] Oct 1 14:16:59 server03 kernel: [<fb1d45b7>] _ZN13HashIPLookup54pushEiP6Packet+0xc7/0x110 [click] Oct 1 14:17:00 server03 kernel: [<fb1c0895>] _ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click] Oct 1 14:17:00 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:17:00 server03 kernel: [<fb1dcd23>] _ZN14HashMACLookup514push_extern_ipEP6Packet+0x1b3/0x4d0 [click] Oct 1 14:17:00 server03 kernel: [<fb1d762e>] _ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click] Oct 1 14:17:00 server03 kernel: [<c05d065b>] tcp_v4_do_rcv+0x22/0x269 Oct 1 14:17:00 server03 kernel: [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848 Oct 1 14:17:00 server03 kernel: [<fb1bfb12>] _ZN11ARPQuerier415handle_responseEP6Packet+0x32/0x8f0 [click] Oct 1 14:17:00 server03 kernel: [<fb21f881>] _ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click] Oct 1 14:17:00 server03 kernel: [<fb1fbfdb>] _ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click] Oct 1 14:17:00 server03 kernel: [<fb192904>] _ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click] Oct 1 14:17:00 server03 kernel: [<fb1c0895>] _ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click] Oct 1 14:17:00 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:17:00 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:17:00 server03 kernel: [<fb1d762e>] _ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click] Oct 1 14:17:00 server03 kernel: [<c05d7777>] arp_rcv+0xd0/0xe1 Oct 1 14:17:00 server03 kernel: [<c05a141d>] __netif_receive_skb+0x3b3/0x3d5 Oct 1 14:17:00 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f Oct 1 14:17:00 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f Oct 1 14:17:00 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3 Oct 1 14:17:00 server03 kernel: [<c0411f41>] smp_apic_timer_interrupt+0x32/0x45 Oct 1 14:17:00 server03 kernel: [<c040470b>] apic_timer_interrupt+0x1f/0x24 Oct 1 14:17:00 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3 Oct 1 14:17:00 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3 Oct 1 14:17:01 server03 kernel: [<fb1fe516>] _ZN10Classifier4pushEiP6Packet+0x76/0xe0 [click] Oct 1 14:17:01 server03 kernel: [<fb155806>] _ZN7Element4pushEiP6Packet+0x26/0x40 [click] Oct 1 14:17:01 server03 kernel: [<fb1ab2a5>] _ZN10FromDevice8run_taskEP4Task+0xa5/0x120 [click] Oct 1 14:17:01 server03 kernel: [<fb16c127>] _ZN12RouterThread6driverEv+0x177/0x290 [click] Oct 1 14:17:01 server03 kernel: [<fb21c822>] _Z11click_schedPv+0x82/0x130 [click] Oct 1 14:17:01 server03 kernel: [<fb21c7a0>] _Z11click_schedPv+0x0/0x130 [click] Oct 1 14:17:01 server03 kernel: [<c0404987>] kernel_thread_helper+0x7/0x10 Oct 1 14:17:01 server03 kernel: ======================= #######################* END OF LOGS #* ##################################################### -- -- Krishna Mohan B -- -- Krishna Mohan B
Hi... On Mon, Oct 10, 2011 at 13:34, sri <bskmohan@gmail.com> wrote:
######################## START OF LOGS ############################ Oct 1 14:16:56 server03 kernel: low stack detected by irq handler Oct 1 14:16:56 server03 kernel: [<c0405e9b>] do_IRQ+0x86/0xc3 Oct 1 14:16:56 server03 kernel: [<c04046e6>] common_interrupt+0x1a/0x20 Oct 1 14:16:56 server03 kernel: [<c05a5c06>] neigh_lookup+0x23/0x7f Oct 1 14:16:56 server03 kernel: [<c05d6800>] arp_bind_neighbour+0x50/0x71 Oct 1 14:16:56 server03 kernel: [<c05b7b44>] rt_intern_hash+0x372/0x480 Oct 1 14:16:56 server03 kernel: [<c05b8352>] __ip_route_output_key+0x700/
Looks like something is exhausting kernel stack way too fast. IIRC in 4K kernel stack scenario, 4K is assigned for normal usage and another 4K for irq. So, IMHO, either something is leaking, or code paths are mixing too much and too fast so they all together exhaust the irq stack. For debugging, I am not sure...maybe ftrace stack usage could help you. Grep your kernel .config and find out everything that contains "stack". Or go to "kernel hacking" section and dig into ftrace section. Good luck.... -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Thanks for the Inputs. As there is not kernel crash, crash files are not create here to debug more. Is there any way to force the kernel to create kernel crash files under /var/crash. This helps more If am not wrong. Regards, Sri On Mon, Oct 10, 2011 at 9:20 PM, Mulyadi Santosa <mulyadi.santosa@gmail.com>wrote:
Hi...
On Mon, Oct 10, 2011 at 13:34, sri <bskmohan@gmail.com> wrote:
######################## START OF LOGS ############################ Oct 1 14:16:56 server03 kernel: low stack detected by irq handler Oct 1 14:16:56 server03 kernel: [<c0405e9b>] do_IRQ+0x86/0xc3 Oct 1 14:16:56 server03 kernel: [<c04046e6>] common_interrupt+0x1a/0x20 Oct 1 14:16:56 server03 kernel: [<c05a5c06>] neigh_lookup+0x23/0x7f Oct 1 14:16:56 server03 kernel: [<c05d6800>]
arp_bind_neighbour+0x50/0x71
Oct 1 14:16:56 server03 kernel: [<c05b7b44>] rt_intern_hash+0x372/0x480 Oct 1 14:16:56 server03 kernel: [<c05b8352>] __ip_route_output_key+0x700/
Looks like something is exhausting kernel stack way too fast. IIRC in 4K kernel stack scenario, 4K is assigned for normal usage and another 4K for irq.
So, IMHO, either something is leaking, or code paths are mixing too much and too fast so they all together exhaust the irq stack.
For debugging, I am not sure...maybe ftrace stack usage could help you. Grep your kernel .config and find out everything that contains "stack". Or go to "kernel hacking" section and dig into ftrace section.
Good luck....
-- regards,
Mulyadi Santosa Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
-- -- Krishna Mohan B
On Mon, Oct 10, 2011 at 11:54 PM, sri <bskmohan@gmail.com> wrote:
Thanks for the Inputs. As there is not kernel crash, crash files are not create here to debug more. Is there any way to force the kernel to create kernel crash files under /var/crash. echo 1 > /proc/sys/kernel/sysrq echo c > /proc/sysrq-trigger
but before doing this, you need to enable kdump to dump the crash image under /var/crash More on kdump and crash utility to debug kernel crash: http://people.redhat.com/anderson/crash_whitepaper/ Thanks, Amit
participants (3)
-
amit mehta -
Mulyadi Santosa -
sri