Hi, Below are the RCU logs, I have tried enabling RCU debugging but didn't recieve much help. root@ls1021aqds:~# root@ls1021aqds:~# INFO: rcu_preempt self-detected stall on CPU { 0} (t=2100 jiffies g=301 c=300 q=1) CPU: 0 PID: 1099 Comm: irq/177-eth0_g0 Not tainted 3.12.19-rt30+ #2 [<800109b1>] (unwind_backtrace+0x1/0x88) from [<8000e697>] (show_stack+0xb/0xc) [<8000e697>] (show_stack+0xb/0xc) from [<803023a3>] (dump_stack+0x4b/0x84) [<803023a3>] (dump_stack+0x4b/0x84) from [<80051c8d>] (rcu_check_callbacks+0x145/0x468) [<80051c8d>] (rcu_check_callbacks+0x145/0x468) from [<800202b7>] (update_process_times+0x2f/0x48) [<800202b7>] (update_process_times+0x2f/0x48) from [<8004829f>] (tick_sched_handle.isra.9+0x2f/0x38) [<8004829f>] (tick_sched_handle.isra.9+0x2f/0x38) from [<800482d5>] (tick_sched_timer+0x2d/0x50) [<800482d5>] (tick_sched_timer+0x2d/0x50) from [<8002cbdf>] (__run_hrtimer.isra.26+0x5f/0xa4) [<8002cbdf>] (__run_hrtimer.isra.26+0x5f/0xa4) from [<8002d319>] (hrtimer_interrupt+0x101/0x268) [<8002d319>] (hrtimer_interrupt+0x101/0x268) from [<80257fef>] (arch_timer_handler_phys+0x1f/0x22) [<80257fef>] (arch_timer_handler_phys+0x1f/0x22) from [<8003f6eb>] (handle_percpu_devid_irq+0x3b/0x50) [<8003f6eb>] (handle_percpu_devid_irq+0x3b/0x50) from [<8003d2bf>] (generic_handle_irq+0x13/0x1c) [<8003d2bf>] (generic_handle_irq+0x13/0x1c) from [<8000cb7d>] (handle_IRQ+0x3d/0x60) [<8000cb7d>] (handle_IRQ+0x3d/0x60) from [<800083df>] (gic_handle_irq+0x2b/0x44) [<800083df>] (gic_handle_irq+0x2b/0x44) from [<80304c9b>] (__irq_svc+0x3b/0x94) Exception stack(0xba9b7c58 to 0xba9b7ca0) 7c40: 804d6400 ba9b7ce8 7c60: 00000000 00000004 ba84be80 ba84be80 804d7268 8050a3c0 be9f2000 00000000 7c80: bf8c9900 be3b6e60 00891000 ba9b7ca0 802a2e61 80303d5a 00030133 ffffffff [<80304c9b>] (__irq_svc+0x3b/0x94) from [<80303d5a>] (rt_spin_lock_slowlock+0x6/0x1a0) [<80303d5a>] (rt_spin_lock_slowlock+0x6/0x1a0) from [<802a2e61>] (ipv4_dst_destroy+0x1d/0x4c) [<802a2e61>] (ipv4_dst_destroy+0x1d/0x4c) from [<80294f73>] (dst_destroy+0x3b/0xa4) [<80294f73>] (dst_destroy+0x3b/0xa4) from [<80295395>] (dst_release+0x41/0x50) [<80295395>] (dst_release+0x41/0x50) from [<80287965>] (skb_release_head_state+0x15/0x70) [<80287965>] (skb_release_head_state+0x15/0x70) from [<802897f7>] (skb_release_all+0x9/0x1a) [<802897f7>] (skb_release_all+0x9/0x1a) from [<80289811>] (__kfree_skb+0x9/0x88) [<80289811>] (__kfree_skb+0x9/0x88) from [<8029e521>] (pfifo_fast_enqueue+0x65/0x78) [<8029e521>] (pfifo_fast_enqueue+0x65/0x78) from [<8029179f>] (dev_queue_xmit+0x1c3/0x310) [<8029179f>] (dev_queue_xmit+0x1c3/0x310) from [<802a86eb>] (ip_finish_output+0x2a7/0x304) [<802a86eb>] (ip_finish_output+0x2a7/0x304) from [<8028f5f3>] (__netif_receive_skb_core+0x2d7/0x338) [<8028f5f3>] (__netif_receive_skb_core+0x2d7/0x338) from [<8028f94d>] (netif_receive_skb+0x3d/0x74) [<8028f94d>] (netif_receive_skb+0x3d/0x74) from [<8028fd87>] (napi_gro_receive+0x2b/0x60) [<8028fd87>] (napi_gro_receive+0x2b/0x60) from [<801f2423>] (gfar_clean_rx_ring+0x38b/0x3d4) [<801f2423>] (gfar_clean_rx_ring+0x38b/0x3d4) from [<801f2483>] (gfar_poll_rx_sq+0x17/0x68) [<801f2483>] (gfar_poll_rx_sq+0x17/0x68) from [<8029071b>] (net_rx_action+0x87/0x120) [<8029071b>] (net_rx_action+0x87/0x120) from [<8001b7f5>] (do_current_softirqs+0x105/0x1cc) [<8001b7f5>] (do_current_softirqs+0x105/0x1cc) from [<8001b8f9>] (local_bh_enable+0x3d/0x60) [<8001b8f9>] (local_bh_enable+0x3d/0x60) from [<8003df2f>] (irq_forced_thread_fn+0x2d/0x30) [<8003df2f>] (irq_forced_thread_fn+0x2d/0x30) from [<8003e09f>] (irq_thread+0x8f/0xf8) [<8003e09f>] (irq_thread+0x8f/0xf8) from [<8002a9ad>] (kthread+0x5d/0x6c) [<8002a9ad>] (kthread+0x5d/0x6c) from [<8000c35d>] (ret_from_fork+0x11/0x34) INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 1, t=2102 jiffies, g=302, c=301, q=1) INFO: Stall ended before state dump start Thanks On Wed, Jan 14, 2015 at 9:50 AM, Vignesh Radhakrishnan < vignesh1192@gmail.com> wrote:
This seems to have hit RCU stalls here which got solved while detecting this. This happens when a grace period is not completed for more than 21 seconds (modifiable timeout from config).
If you are able to reproduce this, can you try the following things :
1. Enable lockdep and use debug_show_all_locks() API inside the stall detection code to see who are holding the read locks and figure out why they are not releasing it 2. Enable RCU_DEBUGGING options from config and get data points from those to see what is happening using ftrace.
Thanks and regards, Vignesh Radhakrishnan
On Wed, Jan 14, 2015 at 9:00 AM, meenakshi aggarwal <meenu23688@gmail.com> wrote:
Hi,
I am receiving following kernel warning while running kernel
“3.12.19-rt30+ #6 SMP PREEMPT RT Tue Jan 13 15:35:39 IST 2015 armv7l GNU/Linux” on ARM v7 platform.
Kindly help.
[ 1044.465822] libphy: mdio@2d24000:02 - Link is Up - 1000/Full
[ 1058.805821] libphy: mdio@2d24000:00 - Link is Up - 1000/Full
[ 1662.764689] ------------[ cut here ]------------
[ 1662.768867] WARNING: CPU: 1 PID: 18 at net/sched/sch_generic.c:279 dev_watchdog+0x137/0x198()
[ 1662.777581] NETDEV WATCHDOG: eth1 (fsl-gianfar): transmit queue 0 timed out
[ 1662.876129] Modules linked in:
[ 1662.885557] CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 3.12.19-rt30+ #1
[ 1662.885609] [<80010991>] (unwind_backtrace+0x1/0x88) from [<8000e677>] (show_stack+0xb/0xc)
[ 1662.885628] [<8000e677>] (show_stack+0xb/0xc) from [<803024d3>] (dump_stack+0x4b/0x84)
[ 1662.885646] [<803024d3>] (dump_stack+0x4b/0x84) from [<80018d4d>] (warn_slowpath_common+0x45/0x64)
[ 1662.885660] [<80018d4d>] (warn_slowpath_common+0x45/0x64) from [<80018d87>] (warn_slowpath_fmt+0x1b/0x24)
[ 1662.885678] [<80018d87>] (warn_slowpath_fmt+0x1b/0x24) from [<8029e5b7>] (dev_watchdog+0x137/0x198)
[ 1662.885697] [<8029e5b7>] (dev_watchdog+0x137/0x198) from [<8001fe53>] (call_timer_fn.isra.32+0x17/0x54)
[ 1662.885712] [<8001fe53>] (call_timer_fn.isra.32+0x17/0x54) from [<8001ff97>] (run_timer_softirq+0x107/0x140)
[ 1662.885725] [<8001ff97>] (run_timer_softirq+0x107/0x140) from [<8001b7ad>] (do_current_softirqs+0x105/0x1cc)
[ 1662.885737] [<8001b7ad>] (do_current_softirqs+0x105/0x1cc) from [<8001b9cb>] (run_ksoftirqd+0x23/0x38)
[ 1662.885754] [<8001b9cb>] (run_ksoftirqd+0x23/0x38) from [<8002fb15>] (smpboot_thread_fn+0x175/0x188)
[ 1662.885769] [<8002fb15>] (smpboot_thread_fn+0x175/0x188) from [<8002a945>] (kthread+0x5d/0x6c)
[ 1662.885784] [<8002a945>] (kthread+0x5d/0x6c) from [<8000c339>] (ret_from_fork+0x11/0x38)
[ 1662.893114] ---[ end trace 0000000000000002 ]---
[ 1682.685516] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 1, t=2102 jiffies, g=489, c=488, q=1155)
[ 1682.685520] INFO: Stall ended before state dump start
Thanks & Regards Meenakshi Aggarwal
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies