Poor network throughput, but double under high cpu load

1

I'm trying to understand a counter-intuitive phenomenon where I get only ~ 400 Mbit/s through a gigabit connection when the CPU is idle, but ~ 850 Mbit/s when the CPU is under full load (stress test utility). I suspect this is a symptom of some misconfiguration, in the end I would like to be able to saturate the link regardless of the CPU usage. The question is: what could cause the poor network performance, and why does it depend on the CPU usage in this strange way?

I'm glad for any hints, thanks for your time!

Systems and hardware

  • Lynx SENYO Mini PC (GE5 PCF55) running up-to-date Arch Linux kernel version 5.0.8 (called the client for reference).
  • NIC is a Realtek RTL8168:

    # lspci -nn | grep Ethernet
    08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 03)
    
  • Connection: CAT 6 patch cable <-> CAT 7 in-wall cable <-> CAT 6 patch field <-> HPE OfficeConnect Switch <-> CAT 6 patch cable <-> supermicro home server (called the server for reference). Another PC gets ~ 950 Mbit/s on the exact same connection to the server with the exact same cables, so I guess this isolates the problem to the client.

Test setup

  • On the server:

    # iperf3 -D -s --bind 192.168.128.1
    
  • On the client while CPUs are idle:

    # iperf3 -c 192.168.128.1
    Connecting to host 192.168.128.1, port 5201
    [  5] local 192.168.128.100 port 60738 connected to 192.168.128.1 port 5201
    [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
    [  5]   0.00-1.00   sec  39.8 MBytes   333 Mbits/sec    0    310 KBytes       
    [  5]   1.00-2.00   sec  40.5 MBytes   339 Mbits/sec    0    322 KBytes       
    [  5]   2.00-3.00   sec  40.3 MBytes   338 Mbits/sec    0    322 KBytes       
    [  5]   3.00-4.00   sec  41.0 MBytes   344 Mbits/sec    0    322 KBytes       
    [  5]   4.00-5.00   sec  36.9 MBytes   310 Mbits/sec    0    322 KBytes       
    [  5]   5.00-6.00   sec  53.1 MBytes   445 Mbits/sec    0    322 KBytes       
    [  5]   6.00-7.00   sec  53.7 MBytes   450 Mbits/sec    0    322 KBytes       
    [  5]   7.00-8.00   sec  54.7 MBytes   459 Mbits/sec    0    338 KBytes       
    [  5]   8.00-9.00   sec  54.0 MBytes   453 Mbits/sec    0    338 KBytes       
    [  5]   9.00-10.00  sec  54.0 MBytes   453 Mbits/sec    0    338 KBytes       
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate         Retr
    [  5]   0.00-10.00  sec   468 MBytes   393 Mbits/sec    0             sender
    [  5]   0.00-10.04  sec   467 MBytes   390 Mbits/sec                  receiver
    
  • On the client while running stress -c 4 (all four CPUs busy on 100%)

    # iperf3 -c 192.168.128.1
    Connecting to host 192.168.128.1, port 5201
    [  5] local 192.168.128.100 port 60742 connected to 192.168.128.1 port 5201
    [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
    [  5]   0.00-1.00   sec   102 MBytes   854 Mbits/sec    0    356 KBytes       
    [  5]   1.00-2.00   sec   101 MBytes   845 Mbits/sec    0    375 KBytes       
    [  5]   2.00-3.00   sec   101 MBytes   846 Mbits/sec    0    392 KBytes       
    [  5]   3.00-4.00   sec   101 MBytes   844 Mbits/sec    0    409 KBytes       
    [  5]   4.00-5.00   sec   100 MBytes   843 Mbits/sec    0    409 KBytes       
    [  5]   5.00-6.00   sec   101 MBytes   843 Mbits/sec    0    409 KBytes       
    [  5]   6.00-7.00   sec   101 MBytes   845 Mbits/sec    0    409 KBytes       
    [  5]   7.00-8.00   sec   100 MBytes   841 Mbits/sec    0    451 KBytes       
    [  5]   8.00-9.00   sec   101 MBytes   843 Mbits/sec    0    451 KBytes       
    [  5]   9.00-10.00  sec   101 MBytes   851 Mbits/sec    0    510 KBytes       
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate         Retr
    [  5]   0.00-10.00  sec  1008 MBytes   846 Mbits/sec    0             sender
    [  5]   0.00-10.04  sec  1005 MBytes   840 Mbits/sec                  receiver
    

Unsuccessful attempts

  • I thought maybe its a frequency scaling issue, but cpupower frequency-set -g performance makes no difference.
  • Reading about issues with my specific NIC, I tried both drivers, the one present in the kernel (r8169) and the one from Realtek (r8168). The general picture is the same, the numbers are a bit different. With the Realtek driver I get 556 Mbits/s (idle CPU) vs 788 Mbits/s (busy CPU).
  • Cross-checked a live Ubuntu 18.04 LTS (kernel 4.18.0) with the same results.
  • Running irqbalance makes no difference.
  • Disabling IPv6 makes no difference.

Additional information (glad to add more on request)

Output from ip

# ip addr show dev enp8s0
2: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:14:0b:81:3f:8a brd ff:ff:ff:ff:ff:ff
    inet 192.168.128.100/24 brd 192.168.128.255 scope global noprefixroute enp8s0
       valid_lft forever preferred_lft forever

Output from ethtool:

# ethtool enp8s0
Settings for enp8s0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full 
                                             100baseT/Half 100baseT/Full 
                                             1000baseT/Full 
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000033 (51)
                               drv probe ifdown ifup
        Link detected: yes

# ethtool -i enp8s0
driver: r8169
version: 
firmware-version: rtl_nic/rtl8168d-1.fw
expansion-rom-version: 
bus-info: 0000:08:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

Interrupts:

# cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
...
 26:     145689          0          0     394154   PCI-MSI 4194304-edge      enp8s0
...

# cat /proc/softirqs 
                    CPU0       CPU1       CPU2       CPU3       
          HI:          1         42         46          1
       TIMER:      11550      32469      10688      11494
      NET_TX:      85221          4          7     189091
      NET_RX:     145692          0         14     394372
       BLOCK:         21        450         39      22765
    IRQ_POLL:          0          0          0          0
     TASKLET:     136092         93         97     384278
       SCHED:      10554      29735       8720       9726
     HRTIMER:          0          0          0          0
         RCU:       6154       8303       5838       6506
linux
networking
cpu-usage
gigabit-ethernet
asked on Super User Apr 22, 2019 by PiQuer

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0