Hi, everyone, I need a help. Anyone who could give me advices or a possible way to find a solution for my problem, would be appreciated. I think I'm getting trouble with a bad use of GSO/GRO. Firstly, I'm putting a piece of the dmesg output that I always get when I make a ping between two terminals. Command used: ping -s 1500 10.0.0.20 PING 10.0.0.20 (10.0.0.20) 1500(1528) bytes of data. 1508 bytes from 10.0.0.20: icmp_seq=3 ttl=64 time=0.161 ms 1508 bytes from 10.0.0.20: icmp_seq=4 ttl=64 time=0.165 ms 1508 bytes from 10.0.0.20: icmp_seq=5 ttl=64 time=0.209 ms dmesg Output: [ 1495.678355] ------------[ cut here ]------------ [ 1495.678364] WARNING: CPU: 0 PID: 1511 at /build/buildd/linux-lts-utopic-3.16.0/net/core/dev.c:2246 skb_warn_bad_offload+0xd0/0xd8() [ 1495.678368] : caps=(0x00000184075b59e9, 0x0000000000000000) *len=1522 data_len=0 gso_size=1460 gso_type=0 ip_summed=0* [ 1495.678370] Modules linked in: openvswitch(OE) veth libcrc32c bnep rfcomm bluetooth 6lowpan_iphc snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm i915 snd_seq_midi snd_seq_midi_event gpio_ich snd_rawmidi snd_seq snd_seq_device coretemp video snd_timer drm_kms_helper kvm_intel drm snd lpc_ich kvm soundcore serio_raw i2c_algo_bit shpchp mac_hid parport_pc ppdev lp parport psmouse pata_acpi r8169 mii [last unloaded: openvswitch] [ 1495.678410] CPU: 0 PID: 1511 Comm: handler43 Tainted: G W OE 3.16.0-30-generic #40~14.04.1-Ubuntu [ 1495.678412] Hardware name: /DG41AN, BIOS ANG4110H.86A.0018.2011.0325.1148 03/25/2011 [ 1495.678415] 00000000 00000000 f6b0d944 c168b423 f6b0d984 f6b0d974 c105c2ae c191ac74 [ 1495.678421] f6b0d9a0 000005e7 c191abfc 000008c6 c168e03e c168e03e f3df4000 f6aef3c0 [ 1495.678428] 000005b4 f6b0d98c c105c303 00000009 f6b0d984 c191ac74 f6b0d9a0 f6b0d9e0 [ 1495.678434] Call Trace: [ 1495.678439] [<c168b423>] dump_stack+0x41/0x52 [ 1495.678444] [<c105c2ae>] warn_slowpath_common+0x7e/0xa0 [ 1495.678449] [<c168e03e>] ? skb_warn_bad_offload+0xd0/0xd8 [ 1495.678452] [<c168e03e>] ? skb_warn_bad_offload+0xd0/0xd8 [ 1495.678456] [<c105c303>] warn_slowpath_fmt+0x33/0x40 [ 1495.678459] [<c168e03e>] skb_warn_bad_offload+0xd0/0xd8 [ 1495.678464] [<c15a52b8>] __skb_gso_segment+0x78/0xb0 [ 1495.678474] [<f8750221>] rpl__skb_gso_segment+0x81/0xc0 [*openvswitch*] [ 1495.678481] [<f874f922>] rpl_dev_queue_xmit+0xa2/0x140 [*openvswitch*] [ 1495.678488] [<f874ef35>] netdev_send+0x65/0xc0 [*openvswitch*] [ 1495.678495] [<f874e6b1>] ovs_vport_send+0x11/0x60 [*openvswitch*] The ping happens after a few seconds, but when I try a ping for the first time, usually 2 or 3 (first) packets are lost and then they keep pinging to each other. The delay is quite high because I'm using 'printks'. Now, let me explain what I've been doing. I'm basically adding a new header (8 bytes size) to the IPv4 packets which goes between L2 and L3, similarly to VLAN. Then, I use Open vSwich (OVS) to keep forwarding my packets. For packet size below MTU size, I have no problem. However, the problem shows up when I try to send a packet having a higher value than the standard MTU (1500 B). One of the questions that I have is, how to use GSO properly? With my understanding, I can segment larger packets before calling *dev_queue_xmit()* or *dev_hard_start_xmit()*, or even before a device driver specific function. OVS has support for earlier versions of the Linux kernel. I'm using int rpl_dev_queue_xmit(struct sk_buff *skb) { #undef dev_queue_xmit int err = -ENOMEM; bool vlan, mpls; vlan = mpls = false; /* Avoid traversing any VLAN tags that are present to determine if * the ethtype is MPLS. Instead compare the mac_len (end of L2) and * skb_network_offset() (beginning of L3) whose inequality will * indicate the presence of an MPLS label stack. */ if (skb->mac_len != skb_network_offset(skb) && !supports_mpls_gso()) mpls = true; if (skb_vlan_tag_present(skb) && !dev_supports_vlan_tx(skb->dev)) vlan = true; if (vlan || mpls) { int features; features = netif_skb_features(skb); if (vlan) { if (!vlan_tso) features &= ~(NETIF_F_TSO | NETIF_F_TSO6 | NETIF_F_UFO | NETIF_F_FSO); skb = vlan_insert_tag_set_proto(skb, skb->vlan_proto, skb_vlan_tag_get(skb)); if (unlikely(!skb)) return err; vlan_set_tci(skb, 0); } /* As of v3.11 the kernel provides an mpls_features field in * struct net_device which allows devices to advertise which * features its supports for MPLS. This value defaults to * NETIF_F_SG and as of v3.19. * * This compatibility code is intended for kernels older * than v3.19 that do not support MPLS GSO and do not * use mpls_features. Thus this code uses NETIF_F_SG * directly in place of mpls_features. */ if (mpls) features &= NETIF_F_SG; if (*netif_needs_gso(skb, features)*) { struct sk_buff *nskb; nskb = *skb_gso_segment*(skb, features); if (!nskb) { if (unlikely(skb_cloned(skb) && pskb_expand_head(skb, 0, 0, GFP_ATOMIC))) goto drop; skb_shinfo(skb)->gso_type &= ~SKB_GSO_DODGY; goto xmit; } if (IS_ERR(nskb)) { err = PTR_ERR(nskb); goto drop; } consume_skb(skb); skb = nskb; do { nskb = skb->next; skb->next = NULL; err = dev_queue_xmit(skb); skb = nskb; } while (skb); return err; } } xmit: return *dev_queue_xmit(skb);* drop: kfree_skb(skb); return err; } My initial idea was to prepare a packet before entering in the code lines in red, but I noticed that netif_needs_gso() requires *skb->gso_size* to be set, if I'm not making a mistake. So, another question is, how to set GSO size properly? At first, I thought I could use any size below MTU, but it seems that is not the case. Besides, for the kernel to properly segment packets according to the Ethertype, I think the kernel must be able to advance the new header, so I modified *skb_mac_gso_segment()* and *skb_network_protocol() *which comes after *skb_gso_segment()* in the kernel [version 3.16 (Ubuntu)]. However, I'm still getting the same dmesg output. As the new header goes in the same place of VLAN, with 8 bytes between L2 and L3, when the packets are segmented, each segment will have L2+New_Header attached to them (until where I know). I checked skb_mac_gso_segment() and I noticed that there is a GSO segment callback for each L3 protocol type. So, I make assumptions that GSO actually happens for protocols above L2, I mean from L3 ahead. My understanding is right? Any help would be very appreciated, Thanks a lot, *--*Airton Ishimori