Not sure I really get 10 Gbit between our servers on the 10 Gbit network

4

We have a 3 Unix (1 Ubuntu 2 Redhat) servers connected via their 10 Gbit network cards to a 10 Gbit switch (our IT person made sure we got the correct cables). We experience some odd behaviour when sending files between the servers (either via scp 10Gbit-ip or a folder mounted via fstab using 10 Gbit ip). The speed is usually far from 10 Gbit but slightly over 1 Gbit. And on one of the server the progess bar is very quick but then gets stuck on 100 % for a long long time. Also true for scp on that server.

To make sure we got Full-Duplex and 10000 Mbit/s I used two different functions and they give conflicting information. Does someone have any ideas of what is going on? mii-tools and ethtool don't give the same information:

em1 = 1 Gbit network (via 10 Gbit port) em2 = 10 Gbit network (via 10 Gbit port)

ifconfig:

[admin@XXXX ~]$ ifconfig 
em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXXX  netmask 255.255.255.0  broadcast XXXXX
        inet6 fe80::eef4:bbff:fed5:c230  prefixlen 64  scopeid 0x20<link>
        ether ec:f4:bb:d5:c2:30  txqueuelen 1000  (Ethernet)
        RX packets 3938043336  bytes 17103106879433 (15.5 TiB)
        RX errors 0  dropped 28  overruns 0  frame 0
        TX packets 8178327495  bytes 8360136815032 (7.6 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

em2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXXXX netmask 255.255.255.0  broadcast XXXX
        inet6 fe80::eef4:bbff:fed5:c232  prefixlen 64  scopeid 0x20<link>
        ether ec:f4:bb:d5:c2:32  txqueuelen 1000  (Ethernet)
        RX packets 3314001  bytes 28089131102 (26.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7632971  bytes 7936298958 (7.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 

mii-tool:

[admin@XXX ~]$ sudo mii-tool em1
[sudo] password for admin: 
SIOCGMIIREG on em1 failed: Invalid argument
SIOCGMIIREG on em1 failed: Invalid argument
SIOCGMIIREG on em1 failed: Invalid argument
em1: 1000 Mbit, half duplex, link ok

[admin@XXX ~]$ sudo mii-tool em2
SIOCGMIIREG on em2 failed: Invalid argument
SIOCGMIIREG on em2 failed: Invalid argument
SIOCGMIIREG on em2 failed: Invalid argument
em2: 1000 Mbit, half duplex, link ok

ethool:

[admin@XXXX ~]$ sudo ethtool em1
Settings for em1:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

[admin@XXXX ~]$ sudo ethtool em2

Settings for em2:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

As you can see everything looks good with ethtool (full dubplex and full speed) but not with mii-tool (1000 Mb/s and half duplex). What could be the reason and potential test / fix?

ubuntu
redhat
unix
network-speed
10gbethernet
asked on Server Fault Jan 16, 2018 by Robin • edited Jan 16, 2018 by Sven

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0