Hi,

Below are the RCU logs, I have tried enabling RCU debugging but didn't recieve much help.



root@ls1021aqds:~#
root@ls1021aqds:~# INFO: rcu_preempt self-detected stall on CPU { 0}  (t=2100 jiffies g=301 c=300 q=1)
CPU: 0 PID: 1099 Comm: irq/177-eth0_g0 Not tainted 3.12.19-rt30+ #2
[<800109b1>] (unwind_backtrace+0x1/0x88) from [<8000e697>] (show_stack+0xb/0xc)
[<8000e697>] (show_stack+0xb/0xc) from [<803023a3>] (dump_stack+0x4b/0x84)
[<803023a3>] (dump_stack+0x4b/0x84) from [<80051c8d>] (rcu_check_callbacks+0x145/0x468)
[<80051c8d>] (rcu_check_callbacks+0x145/0x468) from [<800202b7>] (update_process_times+0x2f/0x48)
[<800202b7>] (update_process_times+0x2f/0x48) from [<8004829f>] (tick_sched_handle.isra.9+0x2f/0x38)
[<8004829f>] (tick_sched_handle.isra.9+0x2f/0x38) from [<800482d5>] (tick_sched_timer+0x2d/0x50)
[<800482d5>] (tick_sched_timer+0x2d/0x50) from [<8002cbdf>] (__run_hrtimer.isra.26+0x5f/0xa4)
[<8002cbdf>] (__run_hrtimer.isra.26+0x5f/0xa4) from [<8002d319>] (hrtimer_interrupt+0x101/0x268)
[<8002d319>] (hrtimer_interrupt+0x101/0x268) from [<80257fef>] (arch_timer_handler_phys+0x1f/0x22)
[<80257fef>] (arch_timer_handler_phys+0x1f/0x22) from [<8003f6eb>] (handle_percpu_devid_irq+0x3b/0x50)
[<8003f6eb>] (handle_percpu_devid_irq+0x3b/0x50) from [<8003d2bf>] (generic_handle_irq+0x13/0x1c)
[<8003d2bf>] (generic_handle_irq+0x13/0x1c) from [<8000cb7d>] (handle_IRQ+0x3d/0x60)
[<8000cb7d>] (handle_IRQ+0x3d/0x60) from [<800083df>] (gic_handle_irq+0x2b/0x44)
[<800083df>] (gic_handle_irq+0x2b/0x44) from [<80304c9b>] (__irq_svc+0x3b/0x94)
Exception stack(0xba9b7c58 to 0xba9b7ca0)
7c40:                                                       804d6400 ba9b7ce8
7c60: 00000000 00000004 ba84be80 ba84be80 804d7268 8050a3c0 be9f2000 00000000
7c80: bf8c9900 be3b6e60 00891000 ba9b7ca0 802a2e61 80303d5a 00030133 ffffffff
[<80304c9b>] (__irq_svc+0x3b/0x94) from [<80303d5a>] (rt_spin_lock_slowlock+0x6/0x1a0)
[<80303d5a>] (rt_spin_lock_slowlock+0x6/0x1a0) from [<802a2e61>] (ipv4_dst_destroy+0x1d/0x4c)
[<802a2e61>] (ipv4_dst_destroy+0x1d/0x4c) from [<80294f73>] (dst_destroy+0x3b/0xa4)
[<80294f73>] (dst_destroy+0x3b/0xa4) from [<80295395>] (dst_release+0x41/0x50)
[<80295395>] (dst_release+0x41/0x50) from [<80287965>] (skb_release_head_state+0x15/0x70)
[<80287965>] (skb_release_head_state+0x15/0x70) from [<802897f7>] (skb_release_all+0x9/0x1a)
[<802897f7>] (skb_release_all+0x9/0x1a) from [<80289811>] (__kfree_skb+0x9/0x88)
[<80289811>] (__kfree_skb+0x9/0x88) from [<8029e521>] (pfifo_fast_enqueue+0x65/0x78)
[<8029e521>] (pfifo_fast_enqueue+0x65/0x78) from [<8029179f>] (dev_queue_xmit+0x1c3/0x310)
[<8029179f>] (dev_queue_xmit+0x1c3/0x310) from [<802a86eb>] (ip_finish_output+0x2a7/0x304)
[<802a86eb>] (ip_finish_output+0x2a7/0x304) from [<8028f5f3>] (__netif_receive_skb_core+0x2d7/0x338)
[<8028f5f3>] (__netif_receive_skb_core+0x2d7/0x338) from [<8028f94d>] (netif_receive_skb+0x3d/0x74)
[<8028f94d>] (netif_receive_skb+0x3d/0x74) from [<8028fd87>] (napi_gro_receive+0x2b/0x60)
[<8028fd87>] (napi_gro_receive+0x2b/0x60) from [<801f2423>] (gfar_clean_rx_ring+0x38b/0x3d4)
[<801f2423>] (gfar_clean_rx_ring+0x38b/0x3d4) from [<801f2483>] (gfar_poll_rx_sq+0x17/0x68)
[<801f2483>] (gfar_poll_rx_sq+0x17/0x68) from [<8029071b>] (net_rx_action+0x87/0x120)
[<8029071b>] (net_rx_action+0x87/0x120) from [<8001b7f5>] (do_current_softirqs+0x105/0x1cc)
[<8001b7f5>] (do_current_softirqs+0x105/0x1cc) from [<8001b8f9>] (local_bh_enable+0x3d/0x60)
[<8001b8f9>] (local_bh_enable+0x3d/0x60) from [<8003df2f>] (irq_forced_thread_fn+0x2d/0x30)
[<8003df2f>] (irq_forced_thread_fn+0x2d/0x30) from [<8003e09f>] (irq_thread+0x8f/0xf8)
[<8003e09f>] (irq_thread+0x8f/0xf8) from [<8002a9ad>] (kthread+0x5d/0x6c)
[<8002a9ad>] (kthread+0x5d/0x6c) from [<8000c35d>] (ret_from_fork+0x11/0x34)
INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 1, t=2102 jiffies, g=302, c=301, q=1)
INFO: Stall ended before state dump start



Thanks

On Wed, Jan 14, 2015 at 9:50 AM, Vignesh Radhakrishnan <vignesh1192@gmail.com> wrote:
This seems to have hit RCU stalls here which got solved while detecting this. This happens when a grace period is not completed for more than 21 seconds (modifiable timeout from config).

If you are able to reproduce this, can you try the following things :

1.  Enable lockdep and use debug_show_all_locks() API inside the stall detection code to see who are holding the read locks and figure out why they are not releasing it
2.  Enable RCU_DEBUGGING options from config and get data points from those to see what is happening using ftrace.

Thanks and regards,
Vignesh Radhakrishnan

On Wed, Jan 14, 2015 at 9:00 AM, meenakshi aggarwal <meenu23688@gmail.com> wrote:
Hi,

 
 

I am receiving following kernel warning while running kernel

“3.12.19-rt30+ #6 SMP PREEMPT RT Tue Jan 13 15:35:39 IST 2015 armv7l GNU/Linux” on ARM v7 platform.

 

Kindly help.

 

 

[ 1044.465822] libphy: mdio@2d24000:02 - Link is Up - 1000/Full

[ 1058.805821] libphy: mdio@2d24000:00 - Link is Up - 1000/Full

[ 1662.764689] ------------[ cut here ]------------

[ 1662.768867] WARNING: CPU: 1 PID: 18 at net/sched/sch_generic.c:279 dev_watchdog+0x137/0x198()

[ 1662.777581] NETDEV WATCHDOG: eth1 (fsl-gianfar): transmit queue 0 timed out

[ 1662.876129] Modules linked in:

[ 1662.885557] CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 3.12.19-rt30+ #1

[ 1662.885609] [<80010991>] (unwind_backtrace+0x1/0x88) from [<8000e677>] (show_stack+0xb/0xc)

[ 1662.885628] [<8000e677>] (show_stack+0xb/0xc) from [<803024d3>] (dump_stack+0x4b/0x84)

[ 1662.885646] [<803024d3>] (dump_stack+0x4b/0x84) from [<80018d4d>] (warn_slowpath_common+0x45/0x64)

[ 1662.885660] [<80018d4d>] (warn_slowpath_common+0x45/0x64) from [<80018d87>] (warn_slowpath_fmt+0x1b/0x24)

[ 1662.885678] [<80018d87>] (warn_slowpath_fmt+0x1b/0x24) from [<8029e5b7>] (dev_watchdog+0x137/0x198)

[ 1662.885697] [<8029e5b7>] (dev_watchdog+0x137/0x198) from [<8001fe53>] (call_timer_fn.isra.32+0x17/0x54)

[ 1662.885712] [<8001fe53>] (call_timer_fn.isra.32+0x17/0x54) from [<8001ff97>] (run_timer_softirq+0x107/0x140)

[ 1662.885725] [<8001ff97>] (run_timer_softirq+0x107/0x140) from [<8001b7ad>] (do_current_softirqs+0x105/0x1cc)

[ 1662.885737] [<8001b7ad>] (do_current_softirqs+0x105/0x1cc) from [<8001b9cb>] (run_ksoftirqd+0x23/0x38)

[ 1662.885754] [<8001b9cb>] (run_ksoftirqd+0x23/0x38) from [<8002fb15>] (smpboot_thread_fn+0x175/0x188)

[ 1662.885769] [<8002fb15>] (smpboot_thread_fn+0x175/0x188) from [<8002a945>] (kthread+0x5d/0x6c)

[ 1662.885784] [<8002a945>] (kthread+0x5d/0x6c) from [<8000c339>] (ret_from_fork+0x11/0x38)

[ 1662.893114] ---[ end trace 0000000000000002 ]---

[ 1682.685516] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 1, t=2102 jiffies, g=489, c=488, q=1155)

[ 1682.685520] INFO: Stall ended before state dump start




Thanks & Regards
Meenakshi Aggarwal

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies




--