Hi, everyone,
I need a help. Anyone who could give me advices or a possible way to find a
solution for my problem, would be appreciated.
I think I'm getting trouble with a bad use of GSO/GRO.
Firstly, I'm putting a piece of the dmesg output that I always get when I
make a ping between two terminals.
Command used:
ping -s 1500 10.0.0.20
PING 10.0.0.20 (10.0.0.20) 1500(1528) bytes of data.
1508 bytes from 10.0.0.20: icmp_seq=3 ttl=64 time=0.161 ms
1508 bytes from 10.0.0.20: icmp_seq=4 ttl=64 time=0.165 ms
1508 bytes from 10.0.0.20: icmp_seq=5 ttl=64 time=0.209 ms
dmesg Output:
[ 1495.678355] ------------[ cut here ]------------
[ 1495.678364] WARNING: CPU: 0 PID: 1511 at
/build/buildd/linux-lts-utopic-3.16.0/net/core/dev.c:2246
skb_warn_bad_offload+0xd0/0xd8()
[ 1495.678368] : caps=(0x00000184075b59e9, 0x0000000000000000) *len=1522
data_len=0 gso_size=1460 gso_type=0 ip_summed=0*
[ 1495.678370] Modules linked in: openvswitch(OE) veth libcrc32c bnep
rfcomm bluetooth 6lowpan_iphc snd_hda_codec_realtek snd_hda_codec_generic
snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm i915
snd_seq_midi snd_seq_midi_event gpio_ich snd_rawmidi snd_seq snd_seq_device
coretemp video snd_timer drm_kms_helper kvm_intel drm snd lpc_ich kvm
soundcore serio_raw i2c_algo_bit shpchp mac_hid parport_pc ppdev lp parport
psmouse pata_acpi r8169 mii [last unloaded: openvswitch]
[ 1495.678410] CPU: 0 PID: 1511 Comm: handler43 Tainted: G W OE
3.16.0-30-generic #40~14.04.1-Ubuntu
[ 1495.678412] Hardware name: /DG41AN, BIOS
ANG4110H.86A.0018.2011.0325.1148 03/25/2011
[ 1495.678415] 00000000 00000000 f6b0d944 c168b423 f6b0d984 f6b0d974
c105c2ae c191ac74
[ 1495.678421] f6b0d9a0 000005e7 c191abfc 000008c6 c168e03e c168e03e
f3df4000 f6aef3c0
[ 1495.678428] 000005b4 f6b0d98c c105c303 00000009 f6b0d984 c191ac74
f6b0d9a0 f6b0d9e0
[ 1495.678434] Call Trace:
[ 1495.678439] [<c168b423>] dump_stack+0x41/0x52
[ 1495.678444] [<c105c2ae>] warn_slowpath_common+0x7e/0xa0
[ 1495.678449] [<c168e03e>] ? skb_warn_bad_offload+0xd0/0xd8
[ 1495.678452] [<c168e03e>] ? skb_warn_bad_offload+0xd0/0xd8
[ 1495.678456] [<c105c303>] warn_slowpath_fmt+0x33/0x40
[ 1495.678459] [<c168e03e>] skb_warn_bad_offload+0xd0/0xd8
[ 1495.678464] [<c15a52b8>] __skb_gso_segment+0x78/0xb0
[ 1495.678474] [<f8750221>] rpl__skb_gso_segment+0x81/0xc0 [*openvswitch*]
[ 1495.678481] [<f874f922>] rpl_dev_queue_xmit+0xa2/0x140 [*openvswitch*]
[ 1495.678488] [<f874ef35>] netdev_send+0x65/0xc0 [*openvswitch*]
[ 1495.678495] [<f874e6b1>] ovs_vport_send+0x11/0x60 [*openvswitch*]
The ping happens after a few seconds, but when I try a ping for the first
time, usually 2 or 3 (first) packets are lost and then they keep pinging to
each other.
The delay is quite high because I'm using 'printks'.
Now, let me explain what I've been doing.
I'm basically adding a new header (8 bytes size) to the IPv4 packets which
goes between L2 and L3, similarly to VLAN.
Then, I use Open vSwich (OVS) to keep forwarding my packets. For packet
size below MTU size, I have no problem. However, the problem shows up when
I try to send a packet having a higher value than the standard MTU (1500 B).
One of the questions that I have is, how to use GSO properly? With my
understanding, I can segment larger packets before calling
*dev_queue_xmit()* or *dev_hard_start_xmit()*, or even before a device
driver specific function.
OVS has support for earlier versions of the Linux kernel. I'm using
int rpl_dev_queue_xmit(struct sk_buff *skb)
{
#undef dev_queue_xmit
int err = -ENOMEM;
bool vlan, mpls;
vlan = mpls = false;
/* Avoid traversing any VLAN tags that are present to determine if
* the ethtype is MPLS. Instead compare the mac_len (end of L2) and
* skb_network_offset() (beginning of L3) whose inequality will
* indicate the presence of an MPLS label stack. */
if (skb->mac_len != skb_network_offset(skb) && !supports_mpls_gso())
mpls = true;
if (skb_vlan_tag_present(skb) && !dev_supports_vlan_tx(skb->dev))
vlan = true;
if (vlan || mpls) {
int features;
features = netif_skb_features(skb);
if (vlan) {
if (!vlan_tso)
features &= ~(NETIF_F_TSO | NETIF_F_TSO6 |
NETIF_F_UFO | NETIF_F_FSO);
skb = vlan_insert_tag_set_proto(skb, skb->vlan_proto,
skb_vlan_tag_get(skb));
if (unlikely(!skb))
return err;
vlan_set_tci(skb, 0);
}
/* As of v3.11 the kernel provides an mpls_features field in
* struct net_device which allows devices to advertise which
* features its supports for MPLS. This value defaults to
* NETIF_F_SG and as of v3.19.
*
* This compatibility code is intended for kernels older
* than v3.19 that do not support MPLS GSO and do not
* use mpls_features. Thus this code uses NETIF_F_SG
* directly in place of mpls_features.
*/
if (mpls)
features &= NETIF_F_SG;
if (*netif_needs_gso(skb, features)*) {
struct sk_buff *nskb;
nskb = *skb_gso_segment*(skb, features);
if (!nskb) {
if (unlikely(skb_cloned(skb) &&
pskb_expand_head(skb, 0, 0, GFP_ATOMIC)))
goto drop;
skb_shinfo(skb)->gso_type &= ~SKB_GSO_DODGY;
goto xmit;
}
if (IS_ERR(nskb)) {
err = PTR_ERR(nskb);
goto drop;
}
consume_skb(skb);
skb = nskb;
do {
nskb = skb->next;
skb->next = NULL;
err = dev_queue_xmit(skb);
skb = nskb;
} while (skb);
return err;
}
}
xmit:
return *dev_queue_xmit(skb);*
drop:
kfree_skb(skb);
return err;
}
My initial idea was to prepare a packet before entering in the code lines
in red, but I noticed that netif_needs_gso() requires *skb->gso_size* to be
set, if I'm not making a mistake.
So, another question is, how to set GSO size properly? At first, I thought
I could use any size below MTU, but it seems that is not the case.
Besides, for the kernel to properly segment packets according to the
Ethertype, I think the kernel must be able to advance the new header, so I
modified *skb_mac_gso_segment()* and
*skb_network_protocol() *which comes after *skb_gso_segment()* in the
kernel [version 3.16 (Ubuntu)]. However, I'm still getting the same dmesg
output.
As the new header goes in the same place of VLAN, with 8 bytes between L2
and L3, when the packets are segmented, each segment will have
L2+New_Header attached to them (until where I know).
I checked skb_mac_gso_segment() and I noticed that there is a GSO segment
callback for each L3 protocol type. So, I make assumptions that GSO
actually happens for protocols above L2, I mean from L3 ahead.
My understanding is right?
Any help would be very appreciated,
Thanks a lot,
*--*Airton Ishimori
