<div dir="ltr">Hi,<div><br></div><div>Below are the RCU logs, I have tried enabling RCU debugging but didn't recieve much help.</div><div><br></div><div><div><br></div><div><br></div><div>root@ls1021aqds:~#</div><div>root@ls1021aqds:~# INFO: rcu_preempt self-detected stall on CPU { 0} (t=2100 jiffies g=301 c=300 q=1)</div><div>CPU: 0 PID: 1099 Comm: irq/177-eth0_g0 Not tainted 3.12.19-rt30+ #2</div><div>[<800109b1>] (unwind_backtrace+0x1/0x88) from [<8000e697>] (show_stack+0xb/0xc)</div><div>[<8000e697>] (show_stack+0xb/0xc) from [<803023a3>] (dump_stack+0x4b/0x84)</div><div>[<803023a3>] (dump_stack+0x4b/0x84) from [<80051c8d>] (rcu_check_callbacks+0x145/0x468)</div><div>[<80051c8d>] (rcu_check_callbacks+0x145/0x468) from [<800202b7>] (update_process_times+0x2f/0x48)</div><div>[<800202b7>] (update_process_times+0x2f/0x48) from [<8004829f>] (tick_sched_handle.isra.9+0x2f/0x38)</div><div>[<8004829f>] (tick_sched_handle.isra.9+0x2f/0x38) from [<800482d5>] (tick_sched_timer+0x2d/0x50)</div><div>[<800482d5>] (tick_sched_timer+0x2d/0x50) from [<8002cbdf>] (__run_hrtimer.isra.26+0x5f/0xa4)</div><div>[<8002cbdf>] (__run_hrtimer.isra.26+0x5f/0xa4) from [<8002d319>] (hrtimer_interrupt+0x101/0x268)</div><div>[<8002d319>] (hrtimer_interrupt+0x101/0x268) from [<80257fef>] (arch_timer_handler_phys+0x1f/0x22)</div><div>[<80257fef>] (arch_timer_handler_phys+0x1f/0x22) from [<8003f6eb>] (handle_percpu_devid_irq+0x3b/0x50)</div><div>[<8003f6eb>] (handle_percpu_devid_irq+0x3b/0x50) from [<8003d2bf>] (generic_handle_irq+0x13/0x1c)</div><div>[<8003d2bf>] (generic_handle_irq+0x13/0x1c) from [<8000cb7d>] (handle_IRQ+0x3d/0x60)</div><div>[<8000cb7d>] (handle_IRQ+0x3d/0x60) from [<800083df>] (gic_handle_irq+0x2b/0x44)</div><div>[<800083df>] (gic_handle_irq+0x2b/0x44) from [<80304c9b>] (__irq_svc+0x3b/0x94)</div><div>Exception stack(0xba9b7c58 to 0xba9b7ca0)</div><div>7c40: 804d6400 ba9b7ce8</div><div>7c60: 00000000 00000004 ba84be80 ba84be80 804d7268 8050a3c0 be9f2000 00000000</div><div>7c80: bf8c9900 be3b6e60 00891000 ba9b7ca0 802a2e61 80303d5a 00030133 ffffffff</div><div>[<80304c9b>] (__irq_svc+0x3b/0x94) from [<80303d5a>] (rt_spin_lock_slowlock+0x6/0x1a0)</div><div>[<80303d5a>] (rt_spin_lock_slowlock+0x6/0x1a0) from [<802a2e61>] (ipv4_dst_destroy+0x1d/0x4c)</div><div>[<802a2e61>] (ipv4_dst_destroy+0x1d/0x4c) from [<80294f73>] (dst_destroy+0x3b/0xa4)</div><div>[<80294f73>] (dst_destroy+0x3b/0xa4) from [<80295395>] (dst_release+0x41/0x50)</div><div>[<80295395>] (dst_release+0x41/0x50) from [<80287965>] (skb_release_head_state+0x15/0x70)</div><div>[<80287965>] (skb_release_head_state+0x15/0x70) from [<802897f7>] (skb_release_all+0x9/0x1a)</div><div>[<802897f7>] (skb_release_all+0x9/0x1a) from [<80289811>] (__kfree_skb+0x9/0x88)</div><div>[<80289811>] (__kfree_skb+0x9/0x88) from [<8029e521>] (pfifo_fast_enqueue+0x65/0x78)</div><div>[<8029e521>] (pfifo_fast_enqueue+0x65/0x78) from [<8029179f>] (dev_queue_xmit+0x1c3/0x310)</div><div>[<8029179f>] (dev_queue_xmit+0x1c3/0x310) from [<802a86eb>] (ip_finish_output+0x2a7/0x304)</div><div>[<802a86eb>] (ip_finish_output+0x2a7/0x304) from [<8028f5f3>] (__netif_receive_skb_core+0x2d7/0x338)</div><div>[<8028f5f3>] (__netif_receive_skb_core+0x2d7/0x338) from [<8028f94d>] (netif_receive_skb+0x3d/0x74)</div><div>[<8028f94d>] (netif_receive_skb+0x3d/0x74) from [<8028fd87>] (napi_gro_receive+0x2b/0x60)</div><div>[<8028fd87>] (napi_gro_receive+0x2b/0x60) from [<801f2423>] (gfar_clean_rx_ring+0x38b/0x3d4)</div><div>[<801f2423>] (gfar_clean_rx_ring+0x38b/0x3d4) from [<801f2483>] (gfar_poll_rx_sq+0x17/0x68)</div><div>[<801f2483>] (gfar_poll_rx_sq+0x17/0x68) from [<8029071b>] (net_rx_action+0x87/0x120)</div><div>[<8029071b>] (net_rx_action+0x87/0x120) from [<8001b7f5>] (do_current_softirqs+0x105/0x1cc)</div><div>[<8001b7f5>] (do_current_softirqs+0x105/0x1cc) from [<8001b8f9>] (local_bh_enable+0x3d/0x60)</div><div>[<8001b8f9>] (local_bh_enable+0x3d/0x60) from [<8003df2f>] (irq_forced_thread_fn+0x2d/0x30)</div><div>[<8003df2f>] (irq_forced_thread_fn+0x2d/0x30) from [<8003e09f>] (irq_thread+0x8f/0xf8)</div><div>[<8003e09f>] (irq_thread+0x8f/0xf8) from [<8002a9ad>] (kthread+0x5d/0x6c)</div><div>[<8002a9ad>] (kthread+0x5d/0x6c) from [<8000c35d>] (ret_from_fork+0x11/0x34)</div><div>INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 1, t=2102 jiffies, g=302, c=301, q=1)</div><div>INFO: Stall ended before state dump start</div></div><div><br></div><div><br></div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 14, 2015 at 9:50 AM, Vignesh Radhakrishnan <span dir="ltr"><<a href="mailto:vignesh1192@gmail.com" target="_blank">vignesh1192@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>This seems to have hit RCU stalls here which got solved while detecting this. This happens when a grace period is not completed for more than 21 seconds (modifiable timeout from config). <br><br>If you are able to reproduce this, can you try the following things :<br><br></div>1. Enable lockdep and use debug_show_all_locks() API inside the stall detection code to see who are holding the read locks and figure out why they are not releasing it<br></div>2. Enable RCU_DEBUGGING options from config and get data points from those to see what is happening using ftrace.<br><br></div>Thanks and regards,<br>Vignesh Radhakrishnan<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Wed, Jan 14, 2015 at 9:00 AM, meenakshi aggarwal <span dir="ltr"><<a href="mailto:meenu23688@gmail.com" target="_blank">meenu23688@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr">Hi,<br><br> <br> <br><br>I am receiving following kernel warning while running kernel<br><br>“3.12.19-rt30+ #6 SMP PREEMPT RT Tue Jan 13 15:35:39 IST 2015 armv7l GNU/Linux” on ARM v7 platform.<br><br> <br><br>Kindly help.<br><br> <br><br> <br><br>[ 1044.465822] libphy: mdio@2d24000:02 - Link is Up - 1000/Full<br><br>[ 1058.805821] libphy: mdio@2d24000:00 - Link is Up - 1000/Full<br><br>[ 1662.764689] ------------[ cut here ]------------<br><br>[ 1662.768867] WARNING: CPU: 1 PID: 18 at net/sched/sch_generic.c:279 dev_watchdog+0x137/0x198()<br><br>[ 1662.777581] NETDEV WATCHDOG: eth1 (fsl-gianfar): transmit queue 0 timed out<br><br>[ 1662.876129] Modules linked in:<br><br>[ 1662.885557] CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 3.12.19-rt30+ #1<br><br>[ 1662.885609] [<80010991>] (unwind_backtrace+0x1/0x88) from [<8000e677>] (show_stack+0xb/0xc)<br><br>[ 1662.885628] [<8000e677>] (show_stack+0xb/0xc) from [<803024d3>] (dump_stack+0x4b/0x84)<br><br>[ 1662.885646] [<803024d3>] (dump_stack+0x4b/0x84) from [<80018d4d>] (warn_slowpath_common+0x45/0x64)<br><br>[ 1662.885660] [<80018d4d>] (warn_slowpath_common+0x45/0x64) from [<80018d87>] (warn_slowpath_fmt+0x1b/0x24)<br><br>[ 1662.885678] [<80018d87>] (warn_slowpath_fmt+0x1b/0x24) from [<8029e5b7>] (dev_watchdog+0x137/0x198)<br><br>[ 1662.885697] [<8029e5b7>] (dev_watchdog+0x137/0x198) from [<8001fe53>] (call_timer_fn.isra.32+0x17/0x54)<br><br>[ 1662.885712] [<8001fe53>] (call_timer_fn.isra.32+0x17/0x54) from [<8001ff97>] (run_timer_softirq+0x107/0x140)<br><br>[ 1662.885725] [<8001ff97>] (run_timer_softirq+0x107/0x140) from [<8001b7ad>] (do_current_softirqs+0x105/0x1cc)<br><br>[ 1662.885737] [<8001b7ad>] (do_current_softirqs+0x105/0x1cc) from [<8001b9cb>] (run_ksoftirqd+0x23/0x38)<br><br>[ 1662.885754] [<8001b9cb>] (run_ksoftirqd+0x23/0x38) from [<8002fb15>] (smpboot_thread_fn+0x175/0x188)<br><br>[ 1662.885769] [<8002fb15>] (smpboot_thread_fn+0x175/0x188) from [<8002a945>] (kthread+0x5d/0x6c)<br><br>[ 1662.885784] [<8002a945>] (kthread+0x5d/0x6c) from [<8000c339>] (ret_from_fork+0x11/0x38)<br><br>[ 1662.893114] ---[ end trace 0000000000000002 ]---<br><br>[ 1682.685516] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 1, t=2102 jiffies, g=489, c=488, q=1155)<br><br>[ 1682.685520] INFO: Stall ended before state dump start<div><br></div><div><br></div><div><br></div><div><br></div><div>Thanks & Regards</div><span><font color="#888888"><div>Meenakshi Aggarwal</div></font></span></div>
<br></div></div>_______________________________________________<br>
Kernelnewbies mailing list<br>
<a href="mailto:Kernelnewbies@kernelnewbies.org" target="_blank">Kernelnewbies@kernelnewbies.org</a><br>
<a href="http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies" target="_blank">http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies</a><br>
<br></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div><a href="http://vigneshradhakrishnan.blogspot.com/" target="_blank">http://vigneshradhakrishnan.blogspot.com/</a><br></div>
</font></span></div>
</blockquote></div><br></div>