10g network cards running at less than 1 gigabit

1

I have installed 10 gigabit fiber-optic cards in two of my servers, but when I test their throughput they run at only around 600Mbps or so:

$ scp random.1G rcm2:/tmp
random.1G 100% 1024MB  73.1MB/s   00:14

My routes look correct. The network drivers and hardware seem to be setup correctly.

Can someone please tell me where I should be looking next?

Here is more information:

Routing

172.16.157.34

$ ip route get 172.16.157.35
172.16.157.35 dev eth1  src 172.16.157.34

172.16.157.35

$ ip route get 172.16.157.34
172.16.157.34 dev eth2  src 172.16.157.35

Hardware/ifconfig

172.16.157.34

$ ifconfig
...
eth2      Link encap:Ethernet  HWaddr 90:e2:ba:38:aa:ee  
          inet addr:172.16.157.34  Bcast:172.16.157.63  Mask:255.255.255.224
...
$ lshw -class network
   ...
   description: Ethernet interface
   product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
   ...
   serial: 90:e2:ba:38:aa:ee
   ...

172.16.157.35

$ ifconfig
...
eth1      Link encap:Ethernet  HWaddr 90:E2:BA:38:AA:82  
          inet addr:172.16.157.35  Bcast:172.16.157.63  Mask:255.255.255.224
...
$ lshw -class network
   ...
   description: Ethernet interface
   product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
   ...
   serial: 90:e2:ba:38:aa:82
   ...

Driver setup

172.16.157.34

$ sudo ethtool eth2 
Settings for eth2:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

172.16.157.35

$ sudo ethtool eth1
[sudo] password for jsp: 
Settings for eth1:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes
routing
10gbethernet
asked on Server Fault Jul 25, 2013 by jsp

1 Answer

2

scp has a large overhead for encrypting the data while in flight. Something thinner like nc will give you better throughput and a more realistic test of the physical capabilities of your server and network.

Also, make sure your bottleneck doesn't exist somewhere else like your disks. Use iperf or something similar to test non-diskbound throughput.

answered on Server Fault Jul 25, 2013 by longneck • edited Jul 25, 2013 by longneck

User contributions licensed under CC BY-SA 3.0