Slow speeds across gigabit intranet in one direction only

The internal network between two of our dedicated servers is slow in one direction and normal in the other direction. Both servers are running CentOS 7.8.2003.

Here I ran an iperf server on the primary node and a client on the secondary node. You can see the transfer speeds are roughly gigabit in one direction and straight up terrible in the other.

------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.156.0.5
TCP window size:  128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  4] local 10.156.0.5 port 5001 connected with 10.156.0.4 port 41260
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.09 GBytes   939 Mbits/sec
------------------------------------------------------------
Client connecting to 10.156.0.4, TCP port 5001
Binding to local address 10.156.0.5
TCP window size:  128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  4] local 10.156.0.5 port 47329 connected with 10.156.0.4 port 5001
[  4]  0.0-10.0 sec  8.62 MBytes  7.23 Mbits/sec


[root@secondary-node ~]# iperf -c 10.156.0.5 -i 2 -w65536 -r
Client connecting to 10.156.0.5, TCP port 5001
TCP window size:  128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  5] local 10.156.0.4 port 41260 connected with 10.156.0.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0- 2.0 sec   224 MBytes   941 Mbits/sec
[  5]  2.0- 4.0 sec   224 MBytes   941 Mbits/sec
[  5]  4.0- 6.0 sec   224 MBytes   941 Mbits/sec
[  5]  6.0- 8.0 sec   224 MBytes   941 Mbits/sec
[  5]  8.0-10.0 sec   224 MBytes   941 Mbits/sec
[  5]  0.0-10.0 sec  1.09 GBytes   941 Mbits/sec
[  4] local 10.156.0.4 port 5001 connected with 10.156.0.5 port 47329
[  4]  0.0- 2.0 sec  1.62 MBytes  6.78 Mbits/sec
[  4]  2.0- 4.0 sec  1.68 MBytes  7.06 Mbits/sec
[  4]  4.0- 6.0 sec  2.35 MBytes  9.86 Mbits/sec
[  4]  6.0- 8.0 sec  2.41 MBytes  10.1 Mbits/sec
[  4]  8.0-10.0 sec   453 KBytes  1.85 Mbits/sec
[  4]  0.0-10.8 sec  8.62 MBytes  6.69 Mbits/sec
[SUM]  0.0-10.8 sec  10.2 MBytes  7.94 Mbits/sec

ethtool eno1 gives the following output for both machines:

Settings for eno1:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: off (auto)
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

Output of mii-tool on both machines is identical as well: eno1: negotiated 1000baseT-FD flow-control, link ok

lspci on the primary:

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:16.1 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
01:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)
02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

lspci on the secondary:

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:16.1 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

lsmod on the primary:

Module                  Size  Used by
rpcsec_gss_krb5        35549  0 
bridge                151336  0 
stp                    12976  1 bridge
llc                    14552  2 stp,bridge
ip_set                 45799  0 
nfnetlink              14519  1 ip_set
ipmi_ssif              29595  0 
intel_powerclamp       14451  0 
coretemp               13444  0 
intel_rapl             19451  0 
iosf_mbi               15582  1 intel_rapl
kvm_intel             188688  0 
kvm                   636965  1 kvm_intel
iTCO_wdt               13572  0 
iTCO_vendor_support    13242  1 iTCO_wdt
irqbypass              13503  1 kvm
crc32_pclmul           13133  0 
ghash_clmulni_intel    13273  0 
aesni_intel           189456  0 
lrw                    13286  1 aesni_intel
gf128mul               15139  1 lrw
glue_helper            13990  1 aesni_intel
ablk_helper            13597  1 aesni_intel
joydev                 17389  0 
pcspkr                 12718  0 
cryptd                 21190  3 ghash_clmulni_intel,aesni_intel,ablk_helper
sg                     40719  0 
i2c_i801               22736  0 
ipmi_si                59965  0 
ipmi_devintf           17459  0 
ie31200_edac           12728  0 
ipmi_msghandler        56728  3 ipmi_ssif,ipmi_devintf,ipmi_si
lpc_ich                21086  0 
nfsd                  351091  13 
mei_me                 32848  0 
mei                    91150  1 mei_me
video                  24538  0 
acpi_pad              116316  0 
auth_rpcgss            59415  2 nfsd,rpcsec_gss_krb5
nfs_acl                12837  1 nfsd
lockd                  98048  1 nfsd
grace                  13515  2 nfsd,lockd
sunrpc                358543  23 nfsd,rpcsec_gss_krb5,auth_rpcgss,lockd,nfs_acl
xfs                   997681  2 
libcrc32c              12644  1 xfs
sd_mod                 46281  4 
crc_t10dif             12912  1 sd_mod
crct10dif_generic      12647  0 
ast                    55467  1 
drm_kms_helper        186531  1 ast
ahci                   34056  0 
syscopyarea            12529  1 drm_kms_helper
sysfillrect            12701  1 drm_kms_helper
sysimgblt              12640  1 drm_kms_helper
fb_sys_fops            12703  1 drm_kms_helper
ttm                    96673  1 ast
libahci                31992  1 ahci
drm                   456166  4 ast,ttm,drm_kms_helper
e1000e                248599  0 
igb                   215727  0 
libata                243094  2 ahci,libahci
megaraid_sas          164492  3 
crct10dif_pclmul       14307  1 
crct10dif_common       12595  3 crct10dif_pclmul,crct10dif_generic,crc_t10dif
crc32c_intel           22094  1 
dca                    15130  1 igb
ptp                    23551  2 igb,e1000e
i2c_algo_bit           13413  2 ast,igb
drm_panel_orientation_quirks    17180  1 drm
pps_core               19057  1 ptp```

lsmod on the secondary:

rpcsec_gss_krb5        35549  0 
auth_rpcgss            59415  1 rpcsec_gss_krb5
nfsv4                 584056  1 
dns_resolver           13140  1 nfsv4
nfs                   262081  2 nfsv4
lockd                  98048  1 nfs
grace                  13515  1 lockd
fscache                64980  2 nfs,nfsv4
bridge                151336  0 
stp                    12976  1 bridge
llc                    14552  2 stp,bridge
ip_set                 45799  0 
nfnetlink              14519  1 ip_set
sunrpc                358543  9 nfs,rpcsec_gss_krb5,auth_rpcgss,lockd,nfsv4
ipmi_ssif              29595  0 
intel_powerclamp       14451  0 
coretemp               13444  0 
intel_rapl             19451  0 
iosf_mbi               15582  1 intel_rapl
kvm_intel             188688  0 
iTCO_wdt               13572  0 
kvm                   636965  1 kvm_intel
iTCO_vendor_support    13242  1 iTCO_wdt
irqbypass              13503  1 kvm
crc32_pclmul           13133  0 
ghash_clmulni_intel    13273  0 
aesni_intel           189456  0 
lrw                    13286  1 aesni_intel
gf128mul               15139  1 lrw
glue_helper            13990  1 aesni_intel
ablk_helper            13597  1 aesni_intel
cryptd                 21190  3 ghash_clmulni_intel,aesni_intel,ablk_helper
i2c_i801               22736  0 
pcspkr                 12718  0 
sg                     40719  0 
joydev                 17389  0 
ipmi_si                59965  0 
ipmi_devintf           17459  0 
ipmi_msghandler        56728  3 ipmi_ssif,ipmi_devintf,ipmi_si
mei_me                 32848  0 
video                  24538  0 
mei                    91150  1 mei_me
lpc_ich                21086  0 
acpi_pad              116316  0 
ie31200_edac           12728  0 
xfs                   997681  2 
libcrc32c              12644  1 xfs
sd_mod                 46281  4 
crc_t10dif             12912  1 sd_mod
crct10dif_generic      12647  0 
ast                    55467  1 
drm_kms_helper        186531  1 ast
syscopyarea            12529  1 drm_kms_helper
sysfillrect            12701  1 drm_kms_helper
sysimgblt              12640  1 drm_kms_helper
fb_sys_fops            12703  1 drm_kms_helper
ttm                    96673  1 ast
ahci                   34056  3 
drm                   456166  4 ast,ttm,drm_kms_helper
libahci                31992  1 ahci
e1000e                248599  0 
igb                   215727  0 
libata                243094  2 ahci,libahci
crct10dif_pclmul       14307  1 
crct10dif_common       12595  3 crct10dif_pclmul,crct10dif_generic,crc_t10dif
crc32c_intel           22094  1 
dca                    15130  1 igb
ptp                    23551  2 igb,e1000e
i2c_algo_bit           13413  2 ast,igb
drm_panel_orientation_quirks    17180  1 drm
pps_core               19057  1 ptp

dmesg | grep eno1 on both machines are the same:

igb 0000:03:00.0 eno1: igb: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

Doesn't seem like a duplex mismatch.. as far as I can tell.

We've had support staff switch out the network cards and the cables, but that didn't help.

There's nothing running on these servers besides nfs/autofs and vsftpd. It's not that iperf is competing with some other process that's pumping data in one direction across the network, because nload shows the same slow speed.

What could the issue be?