<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Hi, everyone, </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">I need a help. Anyone who could give me advices or a possible way to find a solution for my problem, would be appreciated.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-size:small"><span style="font-family:arial,helvetica,sans-serif">I think I'm getting trouble with a bad use of GSO/GRO. </span></div><div class="gmail_default" style="font-size:small"><span style="font-family:arial,helvetica,sans-serif">Firstly, I'm putting a piece of the </span><font face="monospace, monospace">dmesg</font><font face="arial, helvetica, sans-serif"> output that I always get when I make a </font><font face="monospace, monospace">ping</font><font face="arial, helvetica, sans-serif"> between two terminals. </font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif">Command used:</font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace">ping -s 1500 10.0.0.20</font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-size:small"><div class="gmail_default"><font face="monospace, monospace">PING 10.0.0.20 (10.0.0.20) 1500(1528) bytes of data.</font></div><div class="gmail_default"><font face="monospace, monospace">1508 bytes from <a href="http://10.0.0.20">10.0.0.20</a>: icmp_seq=3 ttl=64 time=0.161 ms</font></div><div class="gmail_default"><div class="gmail_default"><font face="monospace, monospace">1508 bytes from <a href="http://10.0.0.20">10.0.0.20</a>: icmp_seq=4 ttl=64 time=0.165 ms</font></div><div class="gmail_default"><font face="monospace, monospace">1508 bytes from <a href="http://10.0.0.20">10.0.0.20</a>: icmp_seq=5 ttl=64 time=0.209 ms</font></div><div><br></div></div><div style="font-family:arial,helvetica,sans-serif"><br></div></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">dmesg Output:</div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-size:small"><div class="gmail_default"><font face="monospace, monospace">[ 1495.678355] ------------[ cut here ]------------</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678364] WARNING: CPU: 0 PID: 1511 at /build/buildd/linux-lts-utopic-3.16.0/net/core/dev.c:2246 skb_warn_bad_offload+0xd0/0xd8()</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678368] : caps=(0x00000184075b59e9, 0x0000000000000000) <b>len=1522 data_len=0 gso_size=1460 gso_type=0 ip_summed=0</b></font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678370] Modules linked in: openvswitch(OE) veth libcrc32c bnep rfcomm bluetooth 6lowpan_iphc snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm i915 snd_seq_midi snd_seq_midi_event gpio_ich snd_rawmidi snd_seq snd_seq_device coretemp video snd_timer drm_kms_helper kvm_intel drm snd lpc_ich kvm soundcore serio_raw i2c_algo_bit shpchp mac_hid parport_pc ppdev lp parport psmouse pata_acpi r8169 mii [last unloaded: openvswitch]</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678410] CPU: 0 PID: 1511 Comm: handler43 Tainted: G W OE 3.16.0-30-generic #40~14.04.1-Ubuntu</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678412] Hardware name: /DG41AN, BIOS ANG4110H.86A.0018.2011.0325.1148 03/25/2011</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678415] 00000000 00000000 f6b0d944 c168b423 f6b0d984 f6b0d974 c105c2ae c191ac74</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678421] f6b0d9a0 000005e7 c191abfc 000008c6 c168e03e c168e03e f3df4000 f6aef3c0</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678428] 000005b4 f6b0d98c c105c303 00000009 f6b0d984 c191ac74 f6b0d9a0 f6b0d9e0</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678434] Call Trace:</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678439] [<c168b423>] dump_stack+0x41/0x52</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678444] [<c105c2ae>] warn_slowpath_common+0x7e/0xa0</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678449] [<c168e03e>] ? skb_warn_bad_offload+0xd0/0xd8</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678452] [<c168e03e>] ? skb_warn_bad_offload+0xd0/0xd8</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678456] [<c105c303>] warn_slowpath_fmt+0x33/0x40</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678459] [<c168e03e>] skb_warn_bad_offload+0xd0/0xd8</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678464] [<c15a52b8>] __skb_gso_segment+0x78/0xb0</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678474] [<f8750221>] rpl__skb_gso_segment+0x81/0xc0 [<b>openvswitch</b>]</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678481] [<f874f922>] rpl_dev_queue_xmit+0xa2/0x140 [<b>openvswitch</b>]</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678488] [<f874ef35>] netdev_send+0x65/0xc0 [<b>openvswitch</b>]</font></div><div class="gmail_default"><font face="monospace, monospace">[ 1495.678495] [<f874e6b1>] ovs_vport_send+0x11/0x60 [<b>openvswitch</b>]</font></div><div style="font-family:arial,helvetica,sans-serif"><br></div></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">The ping happens after a few seconds, but when I try a ping for the first time, usually 2 or 3 (first) packets are lost and then they keep pinging to each other.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">The delay is quite high because I'm using 'printks'.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Now, let me explain what I've been doing.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">I'm basically adding a new header (8 bytes size) to the IPv4 packets which goes between L2 and L3, similarly to VLAN.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Then, I use Open vSwich (OVS) to keep forwarding my packets. For packet size below MTU size, I have no problem. However, the problem shows up when I try to send a packet having a higher value than the standard MTU (1500 B).</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-size:small"><span style="font-family:arial,helvetica,sans-serif">One of the questions that I have is, how to use GSO properly? With my understanding, I can segment larger packets before calling </span><font face="monospace, monospace"><b>dev_queue_xmit()</b></font><font face="arial, helvetica, sans-serif"><b> </b>or </font><font face="monospace, monospace"><b>dev_hard_start_xmit()</b></font><font face="arial, helvetica, sans-serif">, or even before a device driver specific function.</font></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">OVS has support for earlier versions of the Linux kernel. I'm using </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><br></font></div><div class="gmail_default" style=""><div class="gmail_default" style=""><font face="monospace, monospace">int rpl_dev_queue_xmit(struct sk_buff *skb)</font></div><div class="gmail_default" style=""><font face="monospace, monospace">{</font></div><div class="gmail_default" style=""><font face="monospace, monospace">#undef dev_queue_xmit</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>int err = -ENOMEM;</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>bool vlan, mpls;</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>vlan = mpls = false;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span>/* Avoid traversing any VLAN tags that are present to determine if</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * the ethtype is MPLS. Instead compare the mac_len (end of L2) and</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * skb_network_offset() (beginning of L3) whose inequality will</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * indicate the presence of an MPLS label stack. */</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>if (skb->mac_len != skb_network_offset(skb) && !supports_mpls_gso())</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>mpls = true;</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>if (skb_vlan_tag_present(skb) && !dev_supports_vlan_tx(skb->dev))</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>vlan = true;</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>if (vlan || mpls) {</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>int features;</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>features = netif_skb_features(skb);</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>if (vlan) {</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>if (!vlan_tso)</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>features &= ~(NETIF_F_TSO | NETIF_F_TSO6 |</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span> NETIF_F_UFO | NETIF_F_FSO);</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>skb = vlan_insert_tag_set_proto(skb, skb->vlan_proto,</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>skb_vlan_tag_get(skb));</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>if (unlikely(!skb))</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>return err;</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>vlan_set_tci(skb, 0);</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>}</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span>/* As of v3.11 the kernel provides an mpls_features field in</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * struct net_device which allows devices to advertise which</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * features its supports for MPLS. This value defaults to</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * NETIF_F_SG and as of v3.19.</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> *</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * This compatibility code is intended for kernels older</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * than v3.19 that do not support MPLS GSO and do not</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * use mpls_features. Thus this code uses NETIF_F_SG</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> * directly in place of mpls_features.</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#000000"><span class="" style="white-space:pre"> </span> */</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>if (mpls)</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>features &= NETIF_F_SG;</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>if (<b>netif_needs_gso(skb, features)</b>) {</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>struct sk_buff *nskb;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>nskb = <b>skb_gso_segment</b>(skb, features);</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>if (!nskb) {</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>if (unlikely(skb_cloned(skb) &&</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span> pskb_expand_head(skb, 0, 0, GFP_ATOMIC)))</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>goto drop;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>skb_shinfo(skb)->gso_type &= ~SKB_GSO_DODGY;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>goto xmit;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>}</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>if (IS_ERR(nskb)) {</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>err = PTR_ERR(nskb);</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>goto drop;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>}</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>consume_skb(skb);</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>skb = nskb;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>do {</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>nskb = skb->next;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>skb->next = NULL;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>err = dev_queue_xmit(skb);</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>skb = nskb;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>} while (skb);</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>return err;</font></div><div class="gmail_default" style=""><font face="monospace, monospace" color="#990000"><span class="" style="white-space:pre"> </span>}</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>}</font></div><div class="gmail_default" style=""><font face="monospace, monospace">xmit:</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>return <b>dev_queue_xmit(skb);</b></font></div><div class="gmail_default" style=""><font face="monospace, monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace, monospace">drop:</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>kfree_skb(skb);</font></div><div class="gmail_default" style=""><font face="monospace, monospace"><span class="" style="white-space:pre"> </span>return err;</font></div><div class="gmail_default" style=""><font face="monospace, monospace">}</font></div></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-size:small"><span style="font-family:arial,helvetica,sans-serif">My initial idea was to prepare a packet before entering in the code lines in red, but I noticed that </span><font face="monospace, monospace">netif_needs_gso() </font><span style="font-family:arial,helvetica,sans-serif">requires </span><font face="monospace, monospace"><b>skb->gso_size</b></font><font face="arial, helvetica, sans-serif"> to be set, if I'm not making a mistake.</font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif">So, another question is, how to set GSO size properly? At first, I thought I could use any size below MTU, but it seems that is not the case.</font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif">Besides, for the kernel to properly segment packets according to the Ethertype, I think the kernel must be able to advance the new header, so I modified </font><b><font face="monospace, monospace">skb_mac_gso_segment()</font></b><font face="arial, helvetica, sans-serif"> and </font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><b>skb_network_protocol() </b></font><font face="arial, helvetica, sans-serif">which comes after </font><font face="monospace, monospace"><b>skb_gso_segment()</b></font><font face="arial, helvetica, sans-serif"> in the kernel [version 3.16 (Ubuntu)]. However, I'm still getting the same </font><font face="monospace, monospace">dmesg</font><font face="arial, helvetica, sans-serif"> output.</font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif">As the new header goes in the same place of VLAN, with 8 bytes between L2 and L3, when the packets are segmented, each segment will have L2+New_Header attached to them (until where I know).</font></div><div class="gmail_default" style="font-size:small"><span style="font-family:arial,helvetica,sans-serif">I checked skb_mac_gso_segment() and I noticed that there is a GSO segment callback for each L3 protocol type. So, </span><span style="font-family:arial,helvetica,sans-serif">I make assumptions that GSO actually happens for protocols above L2, I mean from L3 ahead. </span></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif">My understanding is right?</font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif">Any help would be very appreciated,</font></div><div class="gmail_default" style="font-size:small"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Thanks a lot,</div><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><i>--<br></i>Airton Ishimori<i><br></i><br><div><div><br></div></div></div></div></div></div></div></div></div></div>
</div>