Network performance deteriorates when uninvolved device on switch is shut down

I am witnessing very counter-intuitive effect in my network setup.

I have a network setup with three devices (let's call them server, laptop and nettop, all running Linux, details below) connected to a gigabit ethernet switch.

I measure network throughput (a) by transferring a file from the server to the laptop using scp and (b) using iperf -s on the server and iperf -c on the laptop.

When I perform the throughput test while all three devices are up and running, I get an ok-ish performance of roughly 40 MB/s (SSH) or 280 Mbits/s (iperf).

However, as soon as I turn off the nettop which isn't involved in the measurement process at all, I get a lousy performance of roughly 3.5 MB/s (SSH) or 58 Mbits/s (iperf).

If it was the other way round, I would try to find out if the nettop is interfering with the transfer somehow. But the fact that performance drops by an order of magnitude when I turn it off leaves me mystified.

This behavior is perfectly reproducible. As soon as the nettop gets pingable, the performance rises immediately. As soon as it is off, the performance drops.

I tried cutting power to the switch for a minute and shuffling around the ethernet ports on the switch with no effect.

So my question is: What could possibly cause the effect I am witnessing? Where could I look in order to narrow down the cause of the problem?

Details on the Setup

Nettop

• Foxconn nT-330i, Intel Atom 330
• Built in Atheros AR8131 Gigabit Ethernet Controller
• Ubuntu 10.04 (lucid), Kernel 2.6.32-29, amd64

Laptop

• Thinkpad R61, Intel Core 2 Duo T8300
• Built in Broadcom BCM5787M Gigabit Ethernet Controller
• Ubuntu 11.10 (oneiric), Kernel 3.0.0-16, amd64

Server

• AMD Athlon 64 X2
• Intel 82574L Gigabit Ethernet Controller (e1000e)
• Debian 6.0.4 (squeeze), Kernel 2.6.32-5, amd64

Switch

• Level One GSW-0806 8-Port Gigabit Switch

The server has two additional network interfaces, one hooked up to a cable modem, the other one to a wifi router.

The server runs a DHCP server which the nettop and laptop acquire their addresses from. It also runs numerous other services, including KVM running two VMs.

IP-Addressing

All devices are in the 10.2.3.0/24 network. Server has 10.2.3.1, laptop 10.2.3.37, nettop 10.2.3.42. The Server interface connected to the Wifi has 10.2.4.1/24. Neither the nettop's nor the laptop's wifi adaptor is enabled.

ethtool output

ethtool shows the following adaptor settings. These outputs are unaffected by turning off and on the nettop.

On the server:

# ethtool eth0
Settings for eth0:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: on
    Supports Wake-on: pumbag
    Wake-on: g
    Current message level: 0x00000001 (1)
    Link detected: yes

On the laptop:

# ethtool eth0
Settings for eth0:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Half 1000baseT/Full 
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Half 1000baseT/Full 
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: g
    Current message level: 0x000000ff (255)
                   drv probe link timer ifdown ifup rx_err tx_err
    Link detected: yes

However, as soon as I turn off the nettop which isn't involved in the measurement process at all, I get a lousy performance of roughly 3.5 MB/s (SSH) or 58 Mbits/s (iperf).

This strongly indicates that the switch (which is a no-brand so-so specimen) drops the entire backplane to 100mbit when you disconnect the nettop.

I agree it is worse than weird - it's unacceptable.

But there you have it.