High Latency during packet transmission

[Vaughn@MSN]

can you post your netstat –s on one of your servers that is having the problem?

From: Michael Krysiak 
Sent: Saturday, June 22, 2013 7:00 AM
To: kernelnewbies at kernelnewbies.org 
Subject: High Latency during packet transmission

I've been trying to identify why we're seeing frequent stalls during packet transmission in our GPFS cluster in the bnx2 driver (as well as other NICs/drivers), but I am at the limit of my current knowledge.  I used perf netdev events (as described in http://lwn.net/Articles/397654/) to measure the tx times, and see spikes such as the following: 

   dev    len      Qdisc               netdevice             free

    em2    98 807740.878085sec        0.002msec             0.061msec
    em2    98 807740.878119sec        0.002msec             0.029msec
    em2    98 807741.140600sec        0.005msec             0.092msec
    em2 65226 807742.763833sec        0.007msec             0.436msec
    em2    66 807727.081712sec        0.001msec         16246.072msec
    em2    66 807740.882741sec        0.001msec          3457.625msec


Based on the source for netdev-times.py, the "free" column is the difference between trace_net_dev_xmit() and trace_kfree_skb() in net/core/dev.c, but I'm not sure how to dig any deeper.  Are there any common causes for this behavior?  What's the best way to further break down the time difference between the xmit and kfree trace points?


--------------------------------------------------------------------------------
_______________________________________________
Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130622/58e1ba7d/attachment.html