Help regarding RT kernel on ARM v7

meenakshi aggarwal meenu23688 at gmail.com
Tue Jan 13 23:45:28 EST 2015 

Hi,

Below are the RCU logs, I have tried enabling RCU debugging but didn't
recieve much help.



root at ls1021aqds:~#
root at ls1021aqds:~# INFO: rcu_preempt self-detected stall on CPU { 0}
 (t=2100 jiffies g=301 c=300 q=1)
CPU: 0 PID: 1099 Comm: irq/177-eth0_g0 Not tainted 3.12.19-rt30+ #2
[<800109b1>] (unwind_backtrace+0x1/0x88) from [<8000e697>]
(show_stack+0xb/0xc)
[<8000e697>] (show_stack+0xb/0xc) from [<803023a3>] (dump_stack+0x4b/0x84)
[<803023a3>] (dump_stack+0x4b/0x84) from [<80051c8d>]
(rcu_check_callbacks+0x145/0x468)
[<80051c8d>] (rcu_check_callbacks+0x145/0x468) from [<800202b7>]
(update_process_times+0x2f/0x48)
[<800202b7>] (update_process_times+0x2f/0x48) from [<8004829f>]
(tick_sched_handle.isra.9+0x2f/0x38)
[<8004829f>] (tick_sched_handle.isra.9+0x2f/0x38) from [<800482d5>]
(tick_sched_timer+0x2d/0x50)
[<800482d5>] (tick_sched_timer+0x2d/0x50) from [<8002cbdf>]
(__run_hrtimer.isra.26+0x5f/0xa4)
[<8002cbdf>] (__run_hrtimer.isra.26+0x5f/0xa4) from [<8002d319>]
(hrtimer_interrupt+0x101/0x268)
[<8002d319>] (hrtimer_interrupt+0x101/0x268) from [<80257fef>]
(arch_timer_handler_phys+0x1f/0x22)
[<80257fef>] (arch_timer_handler_phys+0x1f/0x22) from [<8003f6eb>]
(handle_percpu_devid_irq+0x3b/0x50)
[<8003f6eb>] (handle_percpu_devid_irq+0x3b/0x50) from [<8003d2bf>]
(generic_handle_irq+0x13/0x1c)
[<8003d2bf>] (generic_handle_irq+0x13/0x1c) from [<8000cb7d>]
(handle_IRQ+0x3d/0x60)
[<8000cb7d>] (handle_IRQ+0x3d/0x60) from [<800083df>]
(gic_handle_irq+0x2b/0x44)
[<800083df>] (gic_handle_irq+0x2b/0x44) from [<80304c9b>]
(__irq_svc+0x3b/0x94)
Exception stack(0xba9b7c58 to 0xba9b7ca0)
7c40:                                                       804d6400
ba9b7ce8
7c60: 00000000 00000004 ba84be80 ba84be80 804d7268 8050a3c0 be9f2000
00000000
7c80: bf8c9900 be3b6e60 00891000 ba9b7ca0 802a2e61 80303d5a 00030133
ffffffff
[<80304c9b>] (__irq_svc+0x3b/0x94) from [<80303d5a>]
(rt_spin_lock_slowlock+0x6/0x1a0)
[<80303d5a>] (rt_spin_lock_slowlock+0x6/0x1a0) from [<802a2e61>]
(ipv4_dst_destroy+0x1d/0x4c)
[<802a2e61>] (ipv4_dst_destroy+0x1d/0x4c) from [<80294f73>]
(dst_destroy+0x3b/0xa4)
[<80294f73>] (dst_destroy+0x3b/0xa4) from [<80295395>]
(dst_release+0x41/0x50)
[<80295395>] (dst_release+0x41/0x50) from [<80287965>]
(skb_release_head_state+0x15/0x70)
[<80287965>] (skb_release_head_state+0x15/0x70) from [<802897f7>]
(skb_release_all+0x9/0x1a)
[<802897f7>] (skb_release_all+0x9/0x1a) from [<80289811>]
(__kfree_skb+0x9/0x88)
[<80289811>] (__kfree_skb+0x9/0x88) from [<8029e521>]
(pfifo_fast_enqueue+0x65/0x78)
[<8029e521>] (pfifo_fast_enqueue+0x65/0x78) from [<8029179f>]
(dev_queue_xmit+0x1c3/0x310)
[<8029179f>] (dev_queue_xmit+0x1c3/0x310) from [<802a86eb>]
(ip_finish_output+0x2a7/0x304)
[<802a86eb>] (ip_finish_output+0x2a7/0x304) from [<8028f5f3>]
(__netif_receive_skb_core+0x2d7/0x338)
[<8028f5f3>] (__netif_receive_skb_core+0x2d7/0x338) from [<8028f94d>]
(netif_receive_skb+0x3d/0x74)
[<8028f94d>] (netif_receive_skb+0x3d/0x74) from [<8028fd87>]
(napi_gro_receive+0x2b/0x60)
[<8028fd87>] (napi_gro_receive+0x2b/0x60) from [<801f2423>]
(gfar_clean_rx_ring+0x38b/0x3d4)
[<801f2423>] (gfar_clean_rx_ring+0x38b/0x3d4) from [<801f2483>]
(gfar_poll_rx_sq+0x17/0x68)
[<801f2483>] (gfar_poll_rx_sq+0x17/0x68) from [<8029071b>]
(net_rx_action+0x87/0x120)
[<8029071b>] (net_rx_action+0x87/0x120) from [<8001b7f5>]
(do_current_softirqs+0x105/0x1cc)
[<8001b7f5>] (do_current_softirqs+0x105/0x1cc) from [<8001b8f9>]
(local_bh_enable+0x3d/0x60)
[<8001b8f9>] (local_bh_enable+0x3d/0x60) from [<8003df2f>]
(irq_forced_thread_fn+0x2d/0x30)
[<8003df2f>] (irq_forced_thread_fn+0x2d/0x30) from [<8003e09f>]
(irq_thread+0x8f/0xf8)
[<8003e09f>] (irq_thread+0x8f/0xf8) from [<8002a9ad>] (kthread+0x5d/0x6c)
[<8002a9ad>] (kthread+0x5d/0x6c) from [<8000c35d>] (ret_from_fork+0x11/0x34)
INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 1, t=2102
jiffies, g=302, c=301, q=1)
INFO: Stall ended before state dump start



Thanks

On Wed, Jan 14, 2015 at 9:50 AM, Vignesh Radhakrishnan <
vignesh1192 at gmail.com> wrote:

> This seems to have hit RCU stalls here which got solved while detecting
> this. This happens when a grace period is not completed for more than 21
> seconds (modifiable timeout from config).
>
> If you are able to reproduce this, can you try the following things :
>
> 1.  Enable lockdep and use debug_show_all_locks() API inside the stall
> detection code to see who are holding the read locks and figure out why
> they are not releasing it
> 2.  Enable RCU_DEBUGGING options from config and get data points from
> those to see what is happening using ftrace.
>
> Thanks and regards,
> Vignesh Radhakrishnan
>
> On Wed, Jan 14, 2015 at 9:00 AM, meenakshi aggarwal <meenu23688 at gmail.com>
> wrote:
>
>> Hi,
>>
>>
>>
>>
>> I am receiving following kernel warning while running kernel
>>
>> “3.12.19-rt30+ #6 SMP PREEMPT RT Tue Jan 13 15:35:39 IST 2015 armv7l
>> GNU/Linux” on ARM v7 platform.
>>
>>
>>
>> Kindly help.
>>
>>
>>
>>
>>
>> [ 1044.465822] libphy: mdio at 2d24000:02 - Link is Up - 1000/Full
>>
>> [ 1058.805821] libphy: mdio at 2d24000:00 - Link is Up - 1000/Full
>>
>> [ 1662.764689] ------------[ cut here ]------------
>>
>> [ 1662.768867] WARNING: CPU: 1 PID: 18 at net/sched/sch_generic.c:279
>> dev_watchdog+0x137/0x198()
>>
>> [ 1662.777581] NETDEV WATCHDOG: eth1 (fsl-gianfar): transmit queue 0
>> timed out
>>
>> [ 1662.876129] Modules linked in:
>>
>> [ 1662.885557] CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 3.12.19-rt30+
>> #1
>>
>> [ 1662.885609] [<80010991>] (unwind_backtrace+0x1/0x88) from [<8000e677>]
>> (show_stack+0xb/0xc)
>>
>> [ 1662.885628] [<8000e677>] (show_stack+0xb/0xc) from [<803024d3>]
>> (dump_stack+0x4b/0x84)
>>
>> [ 1662.885646] [<803024d3>] (dump_stack+0x4b/0x84) from [<80018d4d>]
>> (warn_slowpath_common+0x45/0x64)
>>
>> [ 1662.885660] [<80018d4d>] (warn_slowpath_common+0x45/0x64) from
>> [<80018d87>] (warn_slowpath_fmt+0x1b/0x24)
>>
>> [ 1662.885678] [<80018d87>] (warn_slowpath_fmt+0x1b/0x24) from
>> [<8029e5b7>] (dev_watchdog+0x137/0x198)
>>
>> [ 1662.885697] [<8029e5b7>] (dev_watchdog+0x137/0x198) from [<8001fe53>]
>> (call_timer_fn.isra.32+0x17/0x54)
>>
>> [ 1662.885712] [<8001fe53>] (call_timer_fn.isra.32+0x17/0x54) from
>> [<8001ff97>] (run_timer_softirq+0x107/0x140)
>>
>> [ 1662.885725] [<8001ff97>] (run_timer_softirq+0x107/0x140) from
>> [<8001b7ad>] (do_current_softirqs+0x105/0x1cc)
>>
>> [ 1662.885737] [<8001b7ad>] (do_current_softirqs+0x105/0x1cc) from
>> [<8001b9cb>] (run_ksoftirqd+0x23/0x38)
>>
>> [ 1662.885754] [<8001b9cb>] (run_ksoftirqd+0x23/0x38) from [<8002fb15>]
>> (smpboot_thread_fn+0x175/0x188)
>>
>> [ 1662.885769] [<8002fb15>] (smpboot_thread_fn+0x175/0x188) from
>> [<8002a945>] (kthread+0x5d/0x6c)
>>
>> [ 1662.885784] [<8002a945>] (kthread+0x5d/0x6c) from [<8000c339>]
>> (ret_from_fork+0x11/0x38)
>>
>> [ 1662.893114] ---[ end trace 0000000000000002 ]---
>>
>> [ 1682.685516] INFO: rcu_preempt detected stalls on CPUs/tasks: {}
>> (detected by 1, t=2102 jiffies, g=489, c=488, q=1155)
>>
>> [ 1682.685520] INFO: Stall ended before state dump start
>>
>>
>>
>>
>> Thanks & Regards
>> Meenakshi Aggarwal
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
>>
>
>
> --
> http://vigneshradhakrishnan.blogspot.com/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150114/80201fb4/attachment-0001.html

More information about the Kernelnewbies mailing list