net_device: limit rate ot tx packets
Hi All, can someone please explain me, how the kernel handles different transfer rates of different net_devices? Or in other words: How does the systemcall send() know, when to block? An example: cat /dev/zero | pv | nc -u <someip> will show different throughput speeds depending on the network device, the packets are sent over (wlan0 will be slower than eth0). - Can someone point out the location in the linux kernel source, where this is handled? - If I register a net_device. How do I signal to the upper network layers that my driver can only accept packets at a certain rate? I tried stopping the egress queue by calling netif_stop_queue(), but this only has the effect that the queue overruns. I have the feeling that I'm missing out a very vital point on how the kernel's networking subsystem works. Unfortunately, Understanding Linux Network Internals couldn't help me out here. Thanks, Christian
Hi! On 14:11 Sat 13 Apr , christian+kn@wwad.de wrote:
Hi All,
can someone please explain me, how the kernel handles different transfer rates of different net_devices? Or in other words: How does the systemcall send() know, when to block?
An example: cat /dev/zero | pv | nc -u <someip> will show different throughput speeds depending on the network device, the packets are sent over (wlan0 will be slower than eth0).
- Can someone point out the location in the linux kernel source, where this is handled?
- If I register a net_device. How do I signal to the upper network layers that my driver can only accept packets at a certain rate? I tried stopping the egress queue by calling netif_stop_queue(), but this only has the effect that the queue overruns.
I have the feeling that I'm missing out a very vital point on how the kernel's networking subsystem works. Unfortunately, Understanding Linux Network Internals couldn't help me out here.
I think this is what causes the behaviour: 1) If the network device is congested, packets will queue up in the qdisc 2) Socket memory gets used up, semdmsg will probably sleep here: udp_sendmsg ip_make_skb __ip_append_data sock_alloc_send_skb sock_alloc_send_pskb sock_wait_for_wmem 3) When packets are sent, the socket destructor will be called and wake up the sender: sock_wfree() sk->sk_write_space(sk); (which is sock_def_write_space) It would be interesting to see what will happen if the qdisc is smaller than the socket memory... -Michi -- programing a layer 3+4 network protocol for mesh networks see http://michaelblizek.twilightparadox.com
On Sun, 14 Apr 2013 08:15:21 +0200, michi1@michaelblizek.twilightparadox.com said:
It would be interesting to see what will happen if the qdisc is smaller than the socket memory...
As long as the qdisc is able to send at least one MSS at a time, it will eventually clear out the backlog (assuming more isn't added in the meantime). This of course requires you to be using a well-behaved qdisc. But if you're using a broken one, any resulting issues are arguably self-inflicted.
Hi! On 03:45 Sun 14 Apr , Valdis.Kletnieks@vt.edu wrote:
On Sun, 14 Apr 2013 08:15:21 +0200, michi1@michaelblizek.twilightparadox.com said:
It would be interesting to see what will happen if the qdisc is smaller than the socket memory...
As long as the qdisc is able to send at least one MSS at a time, it will eventually clear out the backlog (assuming more isn't added in the meantime). This of course requires you to be using a well-behaved qdisc.
But if you're using a broken one, any resulting issues are arguably self-inflicted.
This is not what I meant. When the qdisc has a size of say 256KB and the socket memory is, say 128kb, the socket memory limit will be reached before the qdisc limit and the socket will sleep. But when the socket memory limit is greater than the qdisc limit, it will be interesting whether the socket still sleeps or starts dropping packets. man send: ENOBUFS The output queue for a network interface was full. This generally indicates that the interface has stopped sending, but may be caused by transient congestion. (Normally, this does not occur in Linux. Packets are just silently dropped when a device queue overflows.) Long live the principle of least surprise... -Michi -- programing a layer 3+4 network protocol for mesh networks see http://michaelblizek.twilightparadox.com
On Sun, 14 Apr 2013 10:09:54 +0200, michi1@michaelblizek.twilightparadox.com said:
This is not what I meant. When the qdisc has a size of say 256KB and the socket memory is, say 128kb, the socket memory limit will be reached before the qdisc limit and the socket will sleep. But when the socket memory limit is greater than the qdisc limit, it will be interesting whether the socket still sleeps or starts dropping packets.
How to figure this out for yourself: Look at net/sched/sch_plug.c, which is a pretty simple qdisc (transmit packets until a plug request is recieved, then queue until unplugged). In particular, look at plug_enqueue() to see what happens when q->limit is exceeded, and plug_init() to see where q->limit is set. Then look at the definition of qdisc_reshape_fail() in include/net/sch_generic.h to figure out what the qdisc returns if q->limit is exceeded. Then go look at net/core/dev.c, in function __dev_xmit_skb(), and watch the variable 'rc'. Now go find the caller of __dev_xmit_skb() and see what it does with that return value. Hope that helps...
How to figure this out for yourself:
Look at net/sched/sch_plug.c, which is a pretty simple qdisc (transmit packets until a plug request is recieved, then queue until unplugged). In particular, look at plug_enqueue() to see what happens when q->limit is exceeded, and plug_init() to see where q->limit is set.
Then look at the definition of qdisc_reshape_fail() in include/net/sch_generic.h to figure out what the qdisc returns if q->limit is exceeded.
Then go look at net/core/dev.c, in function __dev_xmit_skb(), and watch the variable 'rc'.
Now go find the caller of __dev_xmit_skb() and see what it does with that return value.
Hope that helps...
Thank You very much, Michi and Vladis! With Your help, I could figure it out. I tried it out, and as Michi stated: If the tx_queue_len is small, send() will not sleep If the tx_queue_len is large, send() will sleep at some point
participants (3)
-
christian+kn＠wwad.de -
michi1＠michaelblizek.twilightparadox.com -
Valdis.Kletnieks＠vt.edu