Hello Experts, I was using click-1.6.0 on centos 5.3 kernel and recently* ported that to centos5.5 kernel.*. After porting, am getting the (*low stack detected by irq handler) and the *following errors on the console frequently and causing the system unresponsive totally. Even after reboot, this issue is re-occurring some time after every 3 days. System info: kernel: centos 5.5 (2.6.194-el5) kernel Modules using: click router NIC driver: tg3 (broadcom) Note: The above setup was working successfully for years earlier. I suspect tg3 driver is doing something here but not sure of how to confirm this. Am looking for pointers to start with so that I can investigate. I tried to do web search on this but not useful. Also, would like to know any ways/tools to debug using the addresses given in the stack trace. Thanks in advance. ######################## *START OF LOGS* ############################ Oct 1 14:16:56 server03 kernel: low stack detected by irq handler Oct 1 14:16:56 server03 kernel: [<c0405e9b>] do_IRQ+0x86/0xc3 Oct 1 14:16:56 server03 kernel: [<c04046e6>] common_interrupt+0x1a/0x20 Oct 1 14:16:56 server03 kernel: [<c05a5c06>] neigh_lookup+0x23/0x7f Oct 1 14:16:56 server03 kernel: [<c05d6800>] arp_bind_neighbour+0x50/0x71 Oct 1 14:16:56 server03 kernel: [<c05b7b44>] rt_intern_hash+0x372/0x480 Oct 1 14:16:56 server03 kernel: [<c05b8352>] __ip_route_output_key+0x700/ 0x79e Oct 1 14:16:56 server03 kernel: [<c05b83d4>] __ip_route_output_key+0x782/0x79e Oct 1 14:16:57 server03 kernel: [<c05b8403>] ip_route_output_flow+0x13/0x1d4 Oct 1 14:16:57 server03 kernel: [<c05b85cd>] ip_route_output_key+0x9/0xb Oct 1 14:16:57 server03 kernel: [<c05bdd5d>] ip_send_reply+0xf8/0x209 Oct 1 14:16:57 server03 kernel: [<c045f1a6>] cache_alloc_refill+0x122/0x3e4 Oct 1 14:16:57 server03 kernel: [<c05b7c2f>] rt_intern_hash+0x45d/0x480 Oct 1 14:16:57 server03 kernel: [<c05cfc56>] tcp_v4_send_reset+0x104/0x11c Oct 1 14:16:57 server03 kernel: [<c05d0879>] tcp_v4_do_rcv+0x240/0x269 Oct 1 14:16:57 server03 kernel: [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848 Oct 1 14:16:57 server03 kernel: [<c05bb173>] ip_local_deliver+0x157/0x1d0 Oct 1 14:16:57 server03 kernel: [<c05baff2>] ip_rcv+0x3cb/0x3f5 Oct 1 14:16:57 server03 kernel: [<c05a141d>] __netif_receive_skb+0x3b3/0x3d5 Oct 1 14:16:57 server03 kernel: [<fb1b121e>] _ZN6ToHost4pushEiP6Packet+0xee/0x210 [click] Oct 1 14:16:57 server03 kernel: [<fb1ce85d>] _ZN12EnsureEther24pushEiP6Packet+0x1d/0x20 [click] Oct 1 14:16:57 server03 kernel: [<fb1c0895>] _ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click] Oct 1 14:16:57 server03 kernel: [<c05bb173>] ip_local_deliver+0x157/0x1d0 Oct 1 14:16:57 server03 kernel: [<c05baff2>] ip_rcv+0x3cb/0x3f5 Oct 1 14:16:57 server03 kernel: [<fb19e42d>] _ZN4IPRw7Mapping9make_pairEiRK8IPFlowIDS3_iiPS0_S4_+0xad/0xd0 [click] Oct 1 14:16:57 server03 kernel: [<fb19fcfb>] _ZN4IPRw7Pattern14create_mappingEiRK8IPFlowIDiiPNS_7MappingES5_RK7HashMapIS1_S5_E+0x1cb/0x1f0 [click] Oct 1 14:16:57 server03 kernel: [<fb218ce3>] _ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160 [click] Oct 1 14:16:57 server03 kernel: [<fb218d9e>] _ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click] Oct 1 14:16:57 server03 kernel: [<fb1ea535>] _ZN11MacThrottle10l2PushProcEiP6Packet+0x55/0x3b0 [click] Oct 1 14:16:57 server03 kernel: [<fb1c0895>] _ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click] Oct 1 14:16:57 server03 kernel: [<fb218ce3>] _ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160 [click] Oct 1 14:16:57 server03 kernel: [<fb218d9e>] _ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click] Oct 1 14:16:58 server03 kernel: [<fb1eab96>] _ZN11MacThrottle4pushEiP6Packet+0x306/0x1890 [click] Oct 1 14:16:58 server03 kernel: [<c0417f22>] enqueue_task+0x29/0x39 Oct 1 14:16:58 server03 kernel: [<c041805a>] __activate_task+0x4a/0x59 Oct 1 14:16:58 server03 kernel: [<fb1c69d8>] _ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click] Oct 1 14:16:58 server03 kernel: [<fb155806>] _ZN7Element4pushEiP6Packet+0x26/0x40 [click] Oct 1 14:16:58 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:58 server03 kernel: [<fb1ee075>] _ZN17NtlmLoginDetector4pushEiP6Packet+0x65/0x15d0 [click] Oct 1 14:16:58 server03 kernel: [<fb155806>] _ZN7Element4pushEiP6Packet+0x26/0x40 [click] Oct 1 14:16:58 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:58 server03 kernel: [<fb21f881>] _ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click] Oct 1 14:16:58 server03 kernel: [<fb1fbfdb>] _ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click] Oct 1 14:16:58 server03 kernel: [<c05bb173>] ip_local_deliver+0x157/0x1d0 Oct 1 14:16:58 server03 kernel: [<fb1d8e86>] _ZN11HashIPTable4pushEiP6Packet+0xf6/0x2d0 [click] Oct 1 14:16:58 server03 kernel: [<c05a141d>] __netif_receive_skb+0x3b3/0x3d5 Oct 1 14:16:58 server03 kernel: [<fb1c69d8>] _ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click] Oct 1 14:16:58 server03 kernel: [<fb192904>] _ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click] Oct 1 14:16:58 server03 kernel: [<fb1e4517>] _ZN12IPPortLookup4pushEiP6Packet+0x47/0x9e0 [click] Oct 1 14:16:58 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f Oct 1 14:16:58 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3 Oct 1 14:16:58 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f Oct 1 14:16:58 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:58 server03 kernel: [<fb1cd5fc>] _ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click] Oct 1 14:16:59 server03 kernel: [<fb1c69d8>] _ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click] Oct 1 14:16:59 server03 kernel: [<fb1c69d8>] _ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1cdbb8>] _ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click] Oct 1 14:16:59 server03 kernel: [<fb1b1853>] _ZN11ARPQuerier214send_query_forERK9IPAddressitit+0x133/0x3f0 [click] Oct 1 14:16:59 server03 kernel: [<fb1b1dba>] _ZN11ARPQuerier29handle_ipEP6Packet+0x2aa/0x5f0 [click] Oct 1 14:16:59 server03 kernel: [<fb1b31bd>] _ZN11ARPQuerier24pushEiP6Packet+0xcd/0x110 [click] Oct 1 14:16:59 server03 kernel: [<fb1cdbb8>] _ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1cdbb8>] _ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1e234f>] _ZN13IPFilterGroup4pushEiP6Packet+0x19f/0xa80 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb207674>] _ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click] Oct 1 14:16:59 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:16:59 server03 kernel: [<c05a5c59>] neigh_lookup+0x76/0x7f Oct 1 14:16:59 server03 kernel: [<fb1d3492>] _ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click] Oct 1 14:16:59 server03 kernel: [<fb1e66cf>] _ZN15LinearIPFilter24pushEiP6Packet+0x4f/0x3f0 [click] Oct 1 14:16:59 server03 kernel: [<fb1b80e8>] _ZN11ARPQuerier420handle_arp_retaggingEP6Packett+0x58/0x940 [click] Oct 1 14:16:59 server03 kernel: [<c05bf82d>] ip_output+0x1f0/0x227 Oct 1 14:16:59 server03 kernel: [<fb1dd289>] _ZN14HashMACLookup54pushEiP6Packet+0x249/0x890 [click] Oct 1 14:16:59 server03 kernel: [<fb1d45b7>] _ZN13HashIPLookup54pushEiP6Packet+0xc7/0x110 [click] Oct 1 14:17:00 server03 kernel: [<fb1c0895>] _ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click] Oct 1 14:17:00 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:17:00 server03 kernel: [<fb1dcd23>] _ZN14HashMACLookup514push_extern_ipEP6Packet+0x1b3/0x4d0 [click] Oct 1 14:17:00 server03 kernel: [<fb1d762e>] _ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click] Oct 1 14:17:00 server03 kernel: [<c05d065b>] tcp_v4_do_rcv+0x22/0x269 Oct 1 14:17:00 server03 kernel: [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848 Oct 1 14:17:00 server03 kernel: [<fb1bfb12>] _ZN11ARPQuerier415handle_responseEP6Packet+0x32/0x8f0 [click] Oct 1 14:17:00 server03 kernel: [<fb21f881>] _ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click] Oct 1 14:17:00 server03 kernel: [<fb1fbfdb>] _ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click] Oct 1 14:17:00 server03 kernel: [<fb192904>] _ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click] Oct 1 14:17:00 server03 kernel: [<fb1c0895>] _ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click] Oct 1 14:17:00 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:17:00 server03 kernel: [<fb1fb79e>] _ZN6Passer4pushEiP6Packet+0x12e/0x860 [click] Oct 1 14:17:00 server03 kernel: [<fb1d762e>] _ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click] Oct 1 14:17:00 server03 kernel: [<c05d7777>] arp_rcv+0xd0/0xe1 Oct 1 14:17:00 server03 kernel: [<c05a141d>] __netif_receive_skb+0x3b3/0x3d5 Oct 1 14:17:00 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f Oct 1 14:17:00 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f Oct 1 14:17:00 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3 Oct 1 14:17:00 server03 kernel: [<c0411f41>] smp_apic_timer_interrupt+0x32/0x45 Oct 1 14:17:00 server03 kernel: [<c040470b>] apic_timer_interrupt+0x1f/0x24 Oct 1 14:17:00 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3 Oct 1 14:17:00 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3 Oct 1 14:17:01 server03 kernel: [<fb1fe516>] _ZN10Classifier4pushEiP6Packet+0x76/0xe0 [click] Oct 1 14:17:01 server03 kernel: [<fb155806>] _ZN7Element4pushEiP6Packet+0x26/0x40 [click] Oct 1 14:17:01 server03 kernel: [<fb1ab2a5>] _ZN10FromDevice8run_taskEP4Task+0xa5/0x120 [click] Oct 1 14:17:01 server03 kernel: [<fb16c127>] _ZN12RouterThread6driverEv+0x177/0x290 [click] Oct 1 14:17:01 server03 kernel: [<fb21c822>] _Z11click_schedPv+0x82/0x130 [click] Oct 1 14:17:01 server03 kernel: [<fb21c7a0>] _Z11click_schedPv+0x0/0x130 [click] Oct 1 14:17:01 server03 kernel: [<c0404987>] kernel_thread_helper+0x7/0x10 Oct 1 14:17:01 server03 kernel: ======================= #######################* END OF LOGS #* ##################################################### -- -- Krishna Mohan B -- -- Krishna Mohan B