Realtek RTL8111/8168/8411 ethernet card randomly disconnecting
System information:
- kernel: latest OPENVZ kernel (2.6.32-042stab140.4)
- centos version: CentOS release 6.10 (Final)
- network card type: Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
Problem: We have a strange problem with our server. Network is disconnected every now and then, randomly. Sometimes it is up for some days and then network disconnects, sometimes it disconnects dayly, it happened it was up for 2 weeks also, so it is totally unreliable. There are no errors in messages.log or dmesg.log or any other logs. Only thing we see is that the server is disconnected from network. Using a KVM i could log in and restart the network card with commands "ifdown eth0 && ifup eth0". After that the network is back online.
What we tried:
- Installing and updating kernel
- Current kernel uses r8169 driver, and we know that some people solved their network issues by installing r8168 driver. We did that, the problem still exists.
Network config:
/etc/sysconfig/network-scripts/ifcfg-eth0:
DEVICE=eth0
ONBOOT=yes
HWADDR=D4:3D:7E:****
BOOTPROTO=static
IPADDR=144.76.****
NETMASK=255.255.255.255
SCOPE="peer 144.76.****"
IPV6INIT=yes
IPV6ADDR=2a01:4f8:200:****
IPV6_DEFAULTGW=fe80::1
IPV6_DEFAULTDEV=eth0
/etc/sysconfig/network-scripts/route-eth0:
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=144.76.219.1
/etc/resolv.conf and /etc/sysconfig/network is setup correctly too
commands and their output which you might need to help me:
ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000033 (51)
drv probe ifdown ifup
Link detected: yes
ifconfig -a
eth0 Link encap:Ethernet HWaddr D4:3D:7E:EC:F1:64
inet addr:144.76.** Bcast:144.76.** Mask:255.255.255.255
inet6 addr: 2a01:4f8:** Scope:Global
inet6 addr: fe80::d63d:** Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:216416 errors:0 dropped:0 overruns:0 frame:0
TX packets:236235 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:25416051 (24.2 MiB) TX bytes:112658399 (107.4 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:41671 errors:0 dropped:0 overruns:0 frame:0
TX packets:41671 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:7606777 (7.2 MiB) TX bytes:7606777 (7.2 MiB)
venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet6 addr: fe80::1/128 Scope:Link
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:217176 errors:0 dropped:0 overruns:0 frame:0
TX packets:194950 errors:0 dropped:1 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:105725409 (100.8 MiB) TX bytes:20479815 (19.5 MiB)
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether d4:3d:7e:ec:f1:64 brd ff:ff:ff:ff:ff:ff
inet 144.76.** peer 144.76.**/32 brd 144.76.** scope global eth0
inet6 2a01:4f8:** scope global
valid_lft forever preferred_lft forever
inet6 fe80::d63d:7eff:** scope link
valid_lft forever preferred_lft forever
3: venet0: <BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/void
inet6 fe80::1/128 scope link
valid_lft forever preferred_lft forever
lsmod
Module Size Used by
vzethdev 8245 0
pio_kaio 14060 0
pio_nfs 19043 0
pio_direct 30148 2
pfmt_raw 3333 0
pfmt_ploop1 6703 2
ploop 120433 11 pio_kaio,pio_nfs,pio_direct,pfmt_raw,pfmt_ploop1
simfs 5189 0
vzrst 206937 0
nf_nat 23122 1 vzrst
nf_conntrack_ipv4 9650 2 nf_nat
nf_defrag_ipv4 1523 1 nf_conntrack_ipv4
vzcpt 156425 1 vzrst
nfs 449026 3 pio_nfs,vzrst,vzcpt
lockd 78281 2 vzrst,nfs
fscache 61345 1 nfs
auth_rpcgss 46116 1 nfs
nfs_acl 2655 1 nfs
sunrpc 274118 6 pio_nfs,nfs,lockd,auth_rpcgss,nfs_acl
nf_conntrack 81593 4 vzrst,nf_nat,nf_conntrack_ipv4,vzcpt
vziolimit 3775 0
vzdquota 55467 0 [permanent]
ip6t_REJECT 4447 0
ip6table_mangle 3629 0
ip6table_filter 3025 0
ip6_tables 19020 2 ip6table_mangle,ip6table_filter
xt_length 1330 0
xt_hl 1539 0
xt_tcpmss 1615 0
xt_TCPMSS 3549 0
iptable_mangle 3453 0
iptable_filter 2897 2
xt_multiport 2772 2
xt_limit 2126 0
xt_dscp 2065 0
ipt_REJECT 2423 19
ip_tables 18183 2 iptable_mangle,iptable_filter
vzevent 2171 1
vznetdev 18984 4
vzmon 25679 5 vzrst,vzcpt,vznetdev
vzdev 2725 5 vzethdev,vziolimit,vzdquota,vznetdev,vzmon
ipv6 342460 90 vzrst,vzcpt,ip6t_REJECT,ip6table_mangle
ext3 240851 1
jbd 80628 1 ext3
freq_table 4944 0
mperf 1565 0
intel_pstate 14660 0
ppdev 8321 0
parport_pc 22658 0
parport 37265 2 ppdev,parport_pc
iTCO_wdt 7342 0
iTCO_vendor_support 3064 1 iTCO_wdt
r8169 76421 0
mii 5544 1 r8169
i2c_i801 13273 0
i2c_core 29164 1 i2c_i801
sg 29542 0
lpc_ich 13579 0
mfd_core 1935 1 lpc_ich
shpchp 29554 0
ext4 431830 3
jbd2 93732 1 ext4
mbcache 8201 2 ext3,ext4
raid1 33193 3
sd_mod 37126 8
crc_t10dif 1217 1 sd_mod
ahci 43194 6
xhci_hcd 181527 0
video 21990 0
output 2417 1 video
wmi 6287 0
dm_mirror 14904 0
dm_region_hash 12189 1 dm_mirror
dm_log 9938 2 dm_mirror,dm_region_hash
dm_mod 102855 2 dm_mirror,dm_log
lspci -v
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: bus master, fast devsel, latency 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
Kernel driver in use: hsw_uncore
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at f7800000 (64-bit, non-prefetchable) [size=4M]
Memory at e0000000 (64-bit, prefetchable) [size=256M]
I/O ports at f000 [size=64]
Expansion ROM at <unassigned> [disabled]
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [d0] Power Management version 2
Capabilities: [a4] PCI Advanced Features
Kernel modules: i915
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) (prog-if 30 [XHCI])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: bus master, medium devsel, latency 0, IRQ 31
Memory at f7c00000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Kernel driver in use: xhci_hcd
Kernel modules: xhci-hcd
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at f7c16000 (64-bit, non-prefetchable) [size=16]
Capabilities: [50] Power Management version 3
Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) (prog-if 20 [EHCI])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: bus master, medium devsel, latency 0, IRQ 16
Memory at f7c14000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Debug port: BAR=1 offset=00a0
Capabilities: [98] PCI Advanced Features
Kernel driver in use: ehci_hcd
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: df200000-df3fffff
Prefetchable memory behind bridge: 00000000df400000-00000000df5fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Capabilities: [a0] Power Management version 3
Kernel modules: shpchp
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000e000-0000efff
Prefetchable memory behind bridge: 00000000f0000000-00000000f00fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Capabilities: [a0] Power Management version 3
Kernel modules: shpchp
00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5) (prog-if 01 [Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=04, sec-latency=0
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Capabilities: [a0] Power Management version 3
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) (prog-if 20 [EHCI])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: bus master, medium devsel, latency 0, IRQ 23
Memory at f7c13000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Debug port: BAR=1 offset=00a0
Capabilities: [98] PCI Advanced Features
Kernel driver in use: ehci_hcd
00:1f.0 ISA bridge: Intel Corporation H87 Express LPC Controller (rev 05)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: bus master, medium devsel, latency 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) (prog-if 01 [AHCI 1.0])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 32
I/O ports at f0b0 [size=8]
I/O ports at f0a0 [size=4]
I/O ports at f090 [size=8]
I/O ports at f080 [size=4]
I/O ports at f060 [size=32]
Memory at f7c12000 (32-bit, non-prefetchable) [size=2K]
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [70] Power Management version 3
Capabilities: [a8] SATA HBA v1.0
Kernel driver in use: ahci
Kernel modules: ahci
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: medium devsel, IRQ 18
Memory at f7c11000 (64-bit, non-prefetchable) [size=256]
I/O ports at f040 [size=32]
Kernel modules: i2c-i801
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Flags: bus master, fast devsel, latency 0, IRQ 33
I/O ports at e000 [size=256]
Memory at f0004000 (64-bit, prefetchable) [size=4K]
Memory at f0000000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 01
Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
Capabilities: [d0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
Kernel driver in use: r8169
Kernel modules: r8169
03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=03, secondary=04, subordinate=04, sec-latency=32
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express PCI/PCI-X Bridge, MSI 00
Capabilities: [c0] Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7816
Capabilities: [100] Virtual Channel
Kernel modules: shpchp
Even when network was down "cat /sys/class/net/eth0/carrier" echoed "1" and cat /sys/class/net/eth0/operstate echoed "up" so network card was connected....
So what is up ? I read on many forums that this card might have issues and buggy. What do you suggest? Am i right that this problem is definitely hardware issue/ or driver bug/issue. I think the solution would be to get a new attached network card and disable this onboard card. What do you think, thanks in advance for any help!
gergokee2 Answers
" I think the solution would be to get a new attached network card and disable this onboard card. " is absolutely the correct answer. Realtec cards are (and I'm being polite) not server grade technology - if the driver bugginess is not enough reason, consider the extra burden they put on the CPU compared to decent hardware.
I would always replace these cards with an Intel chipset card (but there are plenty of other reputable chipsets) on anything but a Windows desktop.
answered on Server Fault Oct 27, 2019 by 
davidgo
davidgo- Try using
irqbalance, helped me once mitigate a similar issue - If possible, can you check your switch logs? There's a chance that
Port Speed Negotiationisn't set toauto, and if it is, try manually setting it to1Gbps - Stupid but, you never know, check/replace the patch cord. It might look & feel fine, but there CAN be one bent pin which can do this kind of stuff. Source: past experience
answered on Server Fault Oct 27, 2019 by 
falken
User contributions licensed under CC BY-SA 3.0