<div dir="ltr">Sure. Also, here are our current buffer sizes.<div><br></div><div><div>net.core.rmem_max=8388608</div><div>net.core.wmem_max=8388608</div><div>net.ipv4.tcp_rmem=4096 262144 8388608</div><div>net.ipv4.tcp_wmem=4096 262144 8388608</div>
<div>net.core.netdev_max_backlog=20000 # I just increased this from 2500 this morning while testing</div><div>net.ipv4.tcp_mem=752256 1003008 1504512</div><div><br></div><div><br></div><div><div>Ip: </div><div> 15836310622 total packets received</div>
<div> 0 forwarded</div><div> 0 incoming packets discarded</div><div> 15833968779 incoming packets delivered</div><div> 10945745295 requests sent out</div><div> 24 dropped because of missing route</div><div>
Icmp:</div><div> 713392 ICMP messages received</div><div> 27 input ICMP message failed.</div><div> ICMP input histogram:</div><div> destination unreachable: 309</div><div> echo requests: 712035</div>
<div> echo replies: 1023</div><div> timestamp request: 5</div><div> address mask request: 15</div><div> 714011 ICMP messages sent</div><div> 0 ICMP messages failed</div><div> ICMP output histogram:</div>
<div> destination unreachable: 372</div><div> echo request: 1599</div><div> echo replies: 712035</div><div> timestamp replies: 5</div><div>IcmpMsg:</div><div> InType0: 1023</div><div> InType3: 309</div>
<div> InType8: 712035</div><div> InType13: 5</div><div> InType17: 15</div><div> InType37: 5</div><div> OutType0: 712035</div><div> OutType3: 372</div><div> OutType8: 1599</div>
<div> OutType14: 5</div><div>Tcp:</div><div> 600367 active connections openings</div><div> 306733 passive connection openings</div><div> 29 failed connection attempts</div><div> 144696 connection resets received</div>
<div> 37 connections established</div><div> 15833082121 segments received</div><div> 10910239657 segments send out</div><div> 34413098 segments retransmited</div><div> 0 bad segments received.</div><div> 140134 resets sent</div>
<div>Udp:</div><div> 173202 packets received</div><div> 65 packets to unknown port received.</div><div> 0 packet receive errors</div><div> 378536 packets sent</div><div>UdpLite:</div><div>TcpExt: [2/9230]</div>
<div> 6 invalid SYN cookies received</div><div> 5 resets received for embryonic SYN_RECV sockets</div><div> 450342 TCP sockets finished time wait in fast timer</div><div> 129 time wait sockets recycled by time stamp</div>
<div> 1961229 packets rejects in established connections because of timestamp</div><div> 2319351 delayed acks sent</div><div> 4920 delayed acks further delayed because of locked socket</div><div> Quick ack mode was activated 24892595 times</div>
<div> 1803250 packets directly queued to recvmsg prequeue.</div><div> 9817384 packets directly received from backlog</div><div> 7446718877 packets directly received from prequeue</div><div> 7251538419 packets header predicted</div>
<div> 684279 packets header predicted and directly queued to user</div><div> 1691392185 acknowledgments not containing data received</div><div> 8410953710 predicted acknowledgments</div><div> 14145456 times recovered from packet loss due to SACK data</div>
<div> 7 bad SACKs received</div><div> Detected reordering 35 times using FACK</div><div> Detected reordering 926621 times using SACK</div><div> Detected reordering 52787 times using time stamp</div><div> 3572157 congestion windows fully recovered</div>
<div> 7696538 congestion windows partially recovered using Hoe heuristic</div><div> TCPDSACKUndo: 9245647</div><div> 2212 congestion windows recovered after partial ack</div><div> 179224 TCP data loss events</div>
<div> TCPLostRetransmit: 1590</div><div> 6911 timeouts after SACK recovery</div><div> 101 timeouts in loss state</div><div> 22817593 fast retransmits</div><div> 11134591 forward retransmits</div><div> 435599 retransmits in slow start</div>
<div> 3531 other TCP timeouts</div><div> 13687 sack retransmits failed</div><div> 24988980 DSACKs sent for old packets</div><div> 39271 DSACKs sent for out of order packets</div><div> 15785429 DSACKs received</div>
<div> 2567 DSACKs for out of order packets received</div><div> 137242 connections reset due to unexpected data</div><div> 70 connections reset due to early user close</div><div> 1 connections aborted due to timeout</div>
<div> TCPDSACKIgnoredOld: 466</div><div> TCPDSACKIgnoredNoUndo: 743779</div><div> TCPSpuriousRTOs: 72</div><div> TCPSackShifted: 170376999</div><div> TCPSackMerged: 62950559</div><div> TCPSackShiftFallback: 166800040</div>
<div> TCPBacklogDrop: 407</div><div>IpExt:</div><div> InBcastPkts: 5</div><div> InOctets: 64795864053403</div><div> OutOctets: 153981389131849</div><div> InBcastOctets: 2880</div></div></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Sat, Jun 22, 2013 at 9:06 AM, Vaughn@MSN <span dir="ltr"><<a href="mailto:vclinton@msn.com" target="_blank">vclinton@msn.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12pt;font-family:'Calibri'">
<div>can you post your netstat –s on one of your servers that is having the
problem?</div>
<div style="font-style:normal;font-size:small;display:inline;text-decoration:none;font-family:'Calibri';font-weight:normal">
<div style="FONT:10pt tahoma">
<div> </div>
<div style="BACKGROUND:#f5f5f5">
<div><b>From:</b> <a title="mrk191@gmail.com" href="mailto:mrk191@gmail.com" target="_blank">Michael Krysiak</a> </div>
<div><b>Sent:</b> Saturday, June 22, 2013 7:00 AM</div>
<div><b>To:</b> <a title="kernelnewbies@kernelnewbies.org" href="mailto:kernelnewbies@kernelnewbies.org" target="_blank">kernelnewbies@kernelnewbies.org</a>
</div>
<div><b>Subject:</b> High Latency during packet transmission</div></div></div>
<div> </div></div>
<div style="font-style:normal;font-size:small;display:inline;text-decoration:none;font-family:'Calibri';font-weight:normal"><div><div class="h5">
<div dir="ltr"><span style="FONT-SIZE:13px;FONT-FAMILY:arial,sans-serif">I've
been trying to identify why we're seeing frequent stalls during packet
transmission in our GPFS cluster in the bnx2 driver (as well as other
NICs/drivers), but I am at the limit of my current knowledge. I used perf
netdev events (as described in </span><a style="FONT-SIZE:13px;FONT-FAMILY:arial,sans-serif" href="http://lwn.net/Articles/397654/" target="_blank">http://lwn.net/Articles/397654/</a><span style="FONT-SIZE:13px;FONT-FAMILY:arial,sans-serif">) to measure the tx
times, and see spikes such as the following:</span>
<div style="FONT-SIZE:13px;FONT-FAMILY:arial,sans-serif"> </div>
<div style="FONT-SIZE:13px;FONT-FAMILY:arial,sans-serif"><font face="courier new, monospace"> dev
len
Qdisc
netdevice
free<br></font></div>
<div style="FONT-SIZE:13px;FONT-FAMILY:arial,sans-serif">
<div><font face="courier new, monospace">
em2 98
807740.878085sec
0.002msec
0.061msec</font></div>
<div><font face="courier new, monospace">
em2 98
807740.878119sec
0.002msec
0.029msec</font></div>
<div><font face="courier new, monospace">
em2 98
807741.140600sec
0.005msec
0.092msec</font></div>
<div><font face="courier new, monospace"> em2 65226
807742.763833sec
0.007msec
0.436msec</font></div>
<div><font face="courier new, monospace">
em2 66
807727.081712sec
0.001msec
16246.072msec</font></div>
<div><font face="courier new, monospace">
em2 66
807740.882741sec
0.001msec
3457.625msec</font></div></div>
<div style="FONT-SIZE:13px;FONT-FAMILY:arial,sans-serif"> </div>
<div style="FONT-SIZE:13px;FONT-FAMILY:arial,sans-serif"> </div>
<div style="FONT-SIZE:13px;FONT-FAMILY:arial,sans-serif">Based on the source
for netdev-times.py, the "free" column is the difference between
trace_net_dev_xmit() and trace_kfree_skb() in net/core/dev.c, but I'm not sure
how to dig any deeper. Are there any common causes for this
behavior? What's the best way to further break down the time difference
between the xmit and kfree trace points?</div></div>
</div></div><p>
</p><hr>
_______________________________________________<br>Kernelnewbies mailing
list<br><a href="mailto:Kernelnewbies@kernelnewbies.org" target="_blank">Kernelnewbies@kernelnewbies.org</a><br><a href="http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies" target="_blank">http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies</a><br>
<p></p></div></div></div></div>
</blockquote></div><br></div>