<br><br>On Thursday, April 3, 2014, Oleg A. Arkhangelsky <<a href="mailto:sysoleg@yandex.ru">sysoleg@yandex.ru</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello all,<br>
<br>
We've got very strange behavior when testing IP packet forwarding performance<br>
on Sandy Bridge platform (Supermicro X9DRH with the latest BIOS). This is two<br>
socket E5-2690 CPU system. Using different PC we're generating DDoS-like traffic<br>
with rate of about 4.5 million packets per second. Traffic is receiving by two<br>
Intel 82599 NICs and forwarding using the second port of one of this NICs. All<br>
load is evenly distributed among two nodes, so each of 32 CPUs SI usage is<br>
virtually equal.<br>
<br>
Now the strangest part. Few moments after pktgen start on traffic generator PC,<br>
average CPU usage on SB system goes to 30-35%. No packet drops,<br>
no rx_missed_errors, no rx_no_dma_resources. Very nice. But SI usage starts to<br>
decreasing gradually. After about 10 seconds we see ~15% SI average among all<br>
CPUs. Still no packet drops, the same RX rate as in the beginning, RX packet<br>
count is equal to TX packet count. After some time we see that average SI usage<br>
start to go up. Peaked at initial 30-35% it goes down to 15% again. This pattern<br>
is repeated every 80 seconds. Interval is very stable. It is undoubtedly bind<br>
to the test start time, because if we start test, then interrupt it after 10<br>
seconds and start it again we see the same 30% SI peak in a few moments. Then<br>
all timings will be the same.<br>
<br>
During the high load time we see this in "perf top -e cache-misses":<br>
<br>
14017.00 24.9% __netdev_alloc_skb [kernel.kallsyms]<br>
5172.00 9.2% _raw_spin_lock [kernel.kallsyms]<br>
4722.00 8.4% build_skb [kernel.kallsyms]<br>
3603.00 6.4% fib_table_lookup [kernel.kallsyms]<br>
<br>
During the "15% load time" top is different:<br>
<br>
11090.00 20.9% build_skb [kernel.kallsyms]<br>
4879.00 9.2% fib_table_lookup [kernel.kallsyms]<br>
4756.00 9.0% ipt_do_table /lib/modules/3.12.15-BUILD-g2e94e30-dirty/kernel/net/ipv4/netfilter/ip_tables.ko<br>
3042.00 5.7% nf_iterate [kernel.kallsyms]<br>
<br>
And __netdev_alloc_skb is at the end of list:<br>
<br>
911.00 0.5% __netdev_alloc_skb [kernel.kallsyms]<br>
<br>
Some info from "perf stat -a sleep 2":<br>
<br>
15% SI:<br>
28640006291 cycles # 0.447 GHz [83.23%]<br>
38764605205 instructions # 1.35 insns per cycle<br>
<br>
30% SI:<br>
56225552442 cycles # 0.877 GHz [83.23%]<br>
39718182298 instructions # 0.71 insns per cycle<br>
<br>
CPUs never go above C1 state, all cores speed from /proc/cpuinfo is constant at<br>
2899.942 MHz. ASPM is disabled.<br>
<br>
All non-essential userspace apps was explicitly killed for test time, there<br>
was no active cron jobs too. So we should assume no interference with<br>
userspace.<br>
<br>
Kernel version is 3.12.15 (ixgbe 3.21.2), but we have the same behavior with<br>
ancient 2.6.35 (ixgbe 3.10.16). Although on 2.6.35 we sometimes get 160-170<br>
seconds interval and different symbols at the "perf top" output (especially<br>
local_bh_enable() which is completely blows my mind).<br>
<br>
Does anybody have some thoughts about the reasons of this kind of behavior?<br>
Sandy Bridge CPU has many uncore/offcore events, which I can sample, maybe<br>
some of them can shed some light on such behavior?<br>
<br>
</blockquote><div><br></div><div>Is it NUMA system ? This happens when node tries to access memory connected to other CPU. </div><div><br></div><div>Abu Raheda </div>