relationship between cpu_affinity and Packet RX Processing
Hi All, I was reading some stuff on interrupts and irq lines today and I thought I'll expermient with the network rx path. To start with, I've a Virtual Machine running 3.8 linux kernel. My machine has 4 CPU cores, network (eth) interface is driven by pcnet_32 AMD driver and is tied to IRQ line #19. I started some network traffic and I notice that out of those 4 CPUs, only one of them is being used and despite changing the CPU affinity, I still don't see the other cores being used for this network traffic. So based on this behavior(please see the logs below), I've these following queries: i) Does it mean that this network card do not have multiple Rx Queues ? ii) I think all the modern NICs must be implementing multiple Rx Queues and hence Can someone please point me to the simplest of such implemenation in any of the in-tree drivers ? iii) I'm just doing a simple 'ping' to google with of big size packets, As I do not have a peer to use packetgen/netperf/iperf utilities. ref: Comments in double quotes. <logs> $ uname -r 3.8.0-rc6-next-20130208 $ cat /proc/cpuinfo |grep processor processor : 0 processor : 1 processor : 2 processor : 3 "Total 4 cpu cores" $ cat /proc/interrupts|egrep 'eth0|CPU' CPU0 CPU1 CPU2 CPU3 19: 5103 74 33 5 IO-APIC-fasteoi eth0 "IRQ Line #19 for the network device" $ lspci|grep -i ethernet 02:01.0 Ethernet controller: Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE] (rev 10) "AMD NIC" $ lspci -s 02:01.0 -vvv|grep module Kernel modules: pcnet32 $ lsmod|grep pcnet32 pcnet32 40671 0 "driver" # whoami root # cat /proc/irq/19/smp_affinity 03 # cat /proc/irq/19/affinity_hint 00 "I think smp_affinity is a bit map, therefore for all the four cores to be utilized, all 4 bits should be set to 1, which leads to 15(0xf), hence Chaning the cpu affinity" # echo 15 > /proc/irq/19/smp_affinity # cat /proc/irq/19/smp_affinity 15 "started network traffic here and monitoring it" # cat /proc/interrupts|grep eth0 19: 5452 78 33 5 IO-APIC-fasteoi eth0 # cat /proc/interrupts|grep eth0 19: 5488 78 35 5 IO-APIC-fasteoi eth0 # cat /proc/interrupts|grep eth0 19: 5492 78 35 5 IO-APIC-fasteoi eth0 # cat /proc/interrupts|grep eth0 19: 5500 78 35 5 IO-APIC-fasteoi eth0 ................................. .........after some time......... # cat /proc/interrupts|grep eth0 19: 6035 78 42 5 IO-APIC-fasteoi eth0 "Most of the packets are still getting routed through CPU0 *Only*." <logs> -Amit
Hi, Kumar, 1) Regarding rx queues: what does "ls /sys/class/net/eth0/queues" show ? 2) can you try setting affinity for working only with CPU3 and see what happens after some traffic is sent by cat /proc/interrupts? regards, Rami Rosen http://ramirose.wix.com/ramirosen On Tue, Mar 26, 2013 at 8:31 PM, Kumar amit mehta <gmate.amit@gmail.com> wrote:
Hi All,
I was reading some stuff on interrupts and irq lines today and I thought I'll expermient with the network rx path. To start with, I've a Virtual Machine running 3.8 linux kernel. My machine has 4 CPU cores, network (eth) interface is driven by pcnet_32 AMD driver and is tied to IRQ line #19. I started some network traffic and I notice that out of those 4 CPUs, only one of them is being used and despite changing the CPU affinity, I still don't see the other cores being used for this network traffic. So based on this behavior(please see the logs below), I've these following queries: i) Does it mean that this network card do not have multiple Rx Queues ? ii) I think all the modern NICs must be implementing multiple Rx Queues and hence Can someone please point me to the simplest of such implemenation in any of the in-tree drivers ? iii) I'm just doing a simple 'ping' to google with of big size packets, As I do not have a peer to use packetgen/netperf/iperf utilities.
ref: Comments in double quotes. <logs> $ uname -r 3.8.0-rc6-next-20130208
$ cat /proc/cpuinfo |grep processor processor : 0 processor : 1 processor : 2 processor : 3 "Total 4 cpu cores"
$ cat /proc/interrupts|egrep 'eth0|CPU' CPU0 CPU1 CPU2 CPU3 19: 5103 74 33 5 IO-APIC-fasteoi eth0 "IRQ Line #19 for the network device"
$ lspci|grep -i ethernet 02:01.0 Ethernet controller: Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE] (rev 10)
"AMD NIC" $ lspci -s 02:01.0 -vvv|grep module Kernel modules: pcnet32
$ lsmod|grep pcnet32 pcnet32 40671 0 "driver"
# whoami root
# cat /proc/irq/19/smp_affinity 03
# cat /proc/irq/19/affinity_hint 00
"I think smp_affinity is a bit map, therefore for all the four cores to be utilized, all 4 bits should be set to 1, which leads to 15(0xf), hence Chaning the cpu affinity" # echo 15 > /proc/irq/19/smp_affinity # cat /proc/irq/19/smp_affinity 15
"started network traffic here and monitoring it" # cat /proc/interrupts|grep eth0 19: 5452 78 33 5 IO-APIC-fasteoi eth0 # cat /proc/interrupts|grep eth0 19: 5488 78 35 5 IO-APIC-fasteoi eth0 # cat /proc/interrupts|grep eth0 19: 5492 78 35 5 IO-APIC-fasteoi eth0 # cat /proc/interrupts|grep eth0 19: 5500 78 35 5 IO-APIC-fasteoi eth0 ................................. .........after some time......... # cat /proc/interrupts|grep eth0 19: 6035 78 42 5 IO-APIC-fasteoi eth0
"Most of the packets are still getting routed through CPU0 *Only*." <logs>
-Amit
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On Tue, Mar 26, 2013 at 08:35:56PM +0200, Rami Rosen wrote:
1) Regarding rx queues: what does "ls /sys/class/net/eth0/queues" show ?
$ ls /sys/class/net/eth0/queues rx-0 tx-0
2) can you try setting affinity for working only with CPU3 and see what happens after some traffic is sent by cat /proc/interrupts?
# echo 8 > /proc/irq/19/smp_affinity # cat /proc/irq/19/smp_affinity 08 $ cat /proc/interrupts |grep eth0 19: 20635 78 387 1857 IO-APIC-fasteoi eth0 $ cat /proc/interrupts |grep eth0 19: 20635 78 387 3041 IO-APIC-fasteoi eth0 Now only, CPU3 is processing the packets. So, i think, this driver(pcnet32) is not using multiple Rx Queue.
Hi, 1) You can be sure that it does not use more than one rx queue. You can find more info about RX queues in http://media.wix.com/ugd//295986_4ef6dbdf11fd0a7f74f09741b4b5b2ee.pdf (I wrote it, a shameless plug...) 2) Does the irqbalance service, which is common in some distros, is running ? can you try to disable it and run traffic again and check /proc/interrupts afrer some time ? 3) regardless of 2 (meaning not stopping irqbalance if it was active) can you please try "taskset 02 ping www.google.com" and tell us whether interrupts were incremented on the second CPU ? (please send /proc/interrupts | grep eth0 before and after sending, let's say, 5 pings)? Rgs, Rami Rosen http://ramirose.wix.com/ramirosen On Tue, Mar 26, 2013 at 9:05 PM, Kumar amit mehta <gmate.amit@gmail.com> wrote:
On Tue, Mar 26, 2013 at 08:35:56PM +0200, Rami Rosen wrote:
1) Regarding rx queues: what does "ls /sys/class/net/eth0/queues" show ?
$ ls /sys/class/net/eth0/queues rx-0 tx-0
2) can you try setting affinity for working only with CPU3 and see what happens after some traffic is sent by cat /proc/interrupts?
# echo 8 > /proc/irq/19/smp_affinity # cat /proc/irq/19/smp_affinity 08
$ cat /proc/interrupts |grep eth0 19: 20635 78 387 1857 IO-APIC-fasteoi eth0 $ cat /proc/interrupts |grep eth0 19: 20635 78 387 3041 IO-APIC-fasteoi eth0
Now only, CPU3 is processing the packets. So, i think, this driver(pcnet32) is not using multiple Rx Queue.
On Tue, Mar 26, 2013 at 09:31:01PM +0200, Rami Rosen wrote:
Hi,
1) You can be sure that it does not use more than one rx queue.
You can find more info about RX queues in http://media.wix.com/ugd//295986_4ef6dbdf11fd0a7f74f09741b4b5b2ee.pdf
(I wrote it, a shameless plug...)
2) Does the irqbalance service, which is common in some distros, is running ? can you try to disable it and run traffic again and check /proc/interrupts afrer some time ?
3) regardless of 2 (meaning not stopping irqbalance if it was active) can you please try "taskset 02 ping www.google.com" and tell us whether interrupts were incremented on the second CPU ? (please send /proc/interrupts | grep eth0 before and after sending, let's say, 5 pings)?
My apologies for responding after a long time. I lost access to the machine on which this VM was running. Now running this VM on my laptop, which seem to have similar hardware configuration. Please find the details, which you had asked for below:(comment in quotes) amit@ubuntu:~$ cat /proc/cpuinfo |grep processor processor : 0 processor : 1 "2 CPU" amit@ubuntu:~$ cat /proc/interrupts |egrep 'CPU|eth0' CPU0 CPU1 19: 7887 31 IO-APIC-fasteoi eth0 amit@ubuntu:~$ ls -lah /sys/class/net/eth0/device/driver lrwxrwxrwx 1 root root 0 Mar 29 01:13 /sys/class/net/eth0/device/driver -> ../../../../bus/pci/drivers/pcnet32 amit@ubuntu:~$ lsmod|grep -i pcnet32 pcnet32 40671 0 "driver" amit@ubuntu:~$ ls /sys/class/net/eth0/queues rx-0 tx-0 "just one rx-queue" amit@ubuntu:~$ sudo cat /proc/irq/19/smp_affinity 01 "tied to CPU0" "disabling irqbalance" amit@ubuntu:~$ status irqbalance irqbalance start/running, process 1016 amit@ubuntu:~$ sudo stop irqbalance irqbalance stop/waiting amit@ubuntu:~$ taskset 02 ping www.google.com "after sometime" amit@ubuntu:~$ cat /proc/interrupts |egrep 'CPU|eth0' CPU0 CPU1 19: 8046 31 IO-APIC-fasteoi eth0 "No increase in interrupt counts on CPU1" "Now lets use both CPUs" root@ubuntu:/home/amit# echo "3" >/proc/irq/19/smp_affinity root@ubuntu:/home/amit# cat /proc/irq/19/smp_affinity 03 "As before, start ping and expecting both CPUs to be used" amit@ubuntu:~$ taskset 02 ping www.google.com root@ubuntu:/home/amit# cat /proc/interrupts |egrep 'CPU|eth0' CPU0 CPU1 19: 9001 71 IO-APIC-fasteoi eth0 ^^^^ and yes, indeed.. So multiple Rx queues seem to have nothing to do with CPU affinity, as in this case I have a driver implementing a single Rx queue and I see that the rx packet processing happens through both of the CPUs. -Amit
On Tue, Mar 26, 2013 at 09:31:01PM +0200, Rami Rosen wrote:
Hi,
1) You can be sure that it does not use more than one rx queue.
You can find more info about RX queues in http://media.wix.com/ugd//295986_4ef6dbdf11fd0a7f74f09741b4b5b2ee.pdf
Thank you so much for sharing this.
Hi, Stay tuned as I intend to put on the web an expanded document, including documentations about kernel internals (besides networking). Regards, Rami Rosen http://ramirose.wix.com/ramirosen On Fri, Mar 29, 2013 at 4:36 PM, Kumar amit mehta <gmate.amit@gmail.com> wrote:
On Tue, Mar 26, 2013 at 09:31:01PM +0200, Rami Rosen wrote:
Hi,
1) You can be sure that it does not use more than one rx queue.
You can find more info about RX queues in http://media.wix.com/ugd//295986_4ef6dbdf11fd0a7f74f09741b4b5b2ee.pdf
Thank you so much for sharing this.
Thanks a lot for this document Rami! I'm starting to study the kernel and it will greatly help me to figure out how to dig into the source code. Looking forward for the expanded version as well! Best regards, Paulo iPhoneから送信 30/03/2013 05:16、Rami Rosen <roszenrami@gmail.com> のメッセージ:
Hi, Stay tuned as I intend to put on the web an expanded document, including documentations about kernel internals (besides networking).
Regards, Rami Rosen http://ramirose.wix.com/ramirosen
On Fri, Mar 29, 2013 at 4:36 PM, Kumar amit mehta <gmate.amit@gmail.com> wrote:
On Tue, Mar 26, 2013 at 09:31:01PM +0200, Rami Rosen wrote:
Hi,
1) You can be sure that it does not use more than one rx queue.
You can find more info about RX queues in http://media.wix.com/ugd//295986_4ef6dbdf11fd0a7f74f09741b4b5b2ee.pdf
Thank you so much for sharing this.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi Amit, On Mar 26 2013, Kumar amit mehta wrote:
Hi All,
I was reading some stuff on interrupts and irq lines today and I thought I'll expermient with the network rx path. To start with, I've a Virtual Machine running 3.8 linux kernel. My machine has 4 CPU cores, network (eth) interface is driven by pcnet_32 AMD driver and is tied to IRQ line #19. I started some network traffic and I notice that out of those 4 CPUs, only one of them is being used and despite changing the CPU affinity, I still don't see the other cores being used for this network traffic. So based on this behavior(please see the logs below), I've these following queries: i) Does it mean that this network card do not have multiple Rx Queues ?
I don't know for sure what linux does, but the NICs I've seen with multiple queues tend to select queues by hashing incoming packets based on source IP, sourse port, destination IP, destination port and (if TCP) protocol. This is done because you generally want all packets for a given connection in the same queue. For a single ping, always to the same destination, all packets will hash the same, and so wind up in the same queue.
ii) I think all the modern NICs must be implementing multiple Rx Queues and hence Can someone please point me to the simplest of such implemenation in any of the in-tree drivers ? iii) I'm just doing a simple 'ping' to google with of big size packets, As I do not have a peer to use packetgen/netperf/iperf utilities.
-- Arlie
On Tue, Mar 26, 2013 at 11:45:55AM -0700, Arlie Stephens wrote:
I don't know for sure what linux does, but the NICs I've seen with multiple queues tend to select queues by hashing incoming packets based on source IP, sourse port, destination IP, destination port and (if TCP) protocol.
This is done because you generally want all packets for a given connection in the same queue.
For a single ping, always to the same destination, all packets will hash the same, and so wind up in the same queue.
I think what you are referring to is called RSS(Recieve side scalling), which I think is a different feature. IIRC, windows ndis driver implements this feature.
participants (4)
-
Arlie Stephens -
Kumar amit mehta -
Paulo Petruzalek -
Rami Rosen