I have installed 10 gigabit fiber-optic cards in two of my servers, but when I test their throughput they run at only around 600Mbps or so:
$ scp random.1G rcm2:/tmp
random.1G 100% 1024MB 73.1MB/s 00:14
My routes look correct. The network drivers and hardware seem to be setup correctly.
Can someone please tell me where I should be looking next?
Here is more information:
172.16.157.34
$ ip route get 172.16.157.35
172.16.157.35 dev eth1 src 172.16.157.34
172.16.157.35
$ ip route get 172.16.157.34
172.16.157.34 dev eth2 src 172.16.157.35
172.16.157.34
$ ifconfig
...
eth2 Link encap:Ethernet HWaddr 90:e2:ba:38:aa:ee
inet addr:172.16.157.34 Bcast:172.16.157.63 Mask:255.255.255.224
...
$ lshw -class network
...
description: Ethernet interface
product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
...
serial: 90:e2:ba:38:aa:ee
...
172.16.157.35
$ ifconfig
...
eth1 Link encap:Ethernet HWaddr 90:E2:BA:38:AA:82
inet addr:172.16.157.35 Bcast:172.16.157.63 Mask:255.255.255.224
...
$ lshw -class network
...
description: Ethernet interface
product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
...
serial: 90:e2:ba:38:aa:82
...
172.16.157.34
$ sudo ethtool eth2
Settings for eth2:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
172.16.157.35
$ sudo ethtool eth1
[sudo] password for jsp:
Settings for eth1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
scp
has a large overhead for encrypting the data while in flight. Something thinner like nc
will give you better throughput and a more realistic test of the physical capabilities of your server and network.
Also, make sure your bottleneck doesn't exist somewhere else like your disks. Use iperf or something similar to test non-diskbound throughput.
User contributions licensed under CC BY-SA 3.0