Active CARP network loop

2

I have to setup, no matter what, a cluster of 3 active OpenBSD 5.5 with CARP on 3 separate ESXi 5.5 hosts.

I think I'm close to the end but I have this message in /var/log/messages : carp0: replay or network loop detected

Here is my configuration :


On every ESXi

Standard Vswitchs, no VDS Promiscuous mode, mac changing and packet forge ON on every vswitch net.reversePathFwsCheckPromisc=1


NODE 1


/etc/hostname.em0 
inet 172.0.0.19 255.255.0.0 NONE 

/etc/hostname.carp0 
inet 172.0.0.16 255.255.0.0 172.0.255.255 balancing ip-unicast carpnodes 1:0,2:100,3:50 pass "password" carpdev em0 

/etc/sysctl.conf 
net.inet.carp.allow=1 
net.inet.carp.preempt=1 
net.inet.carp.log=2 

/etc/pf.conf 
pass quick on em0 inet proto carp from any to any 

ifconfig carp0 
carp0: flags=8843 mtu 1500 
        lladdr 00:00:5e:00:01:01 
        priority: 0 
        carp: carpdev em0 advbase 1 balancing ip-unicast 
                state MASTER vhid 1 advskew 0 
                state BACKUP vhid 2 advskew 100 
                state BACKUP vhid 3 advskew 50 
        groups: carp 
        status: master 
        inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x4 
        inet 172.0.0.16 netmask 0xffff0000 broadcast 172.0.255.255 

NODE 2


/etc/hostname.em0 
inet 172.0.0.20 255.255.0.0 NONE 

/etc/hostname.carp0 
inet 172.0.0.16 255.255.0.0 172.0.255.255 balancing ip-unicast carpnodes 1:50,2:0,3:100 pass "password" carpdev em0 

/etc/sysctl.conf 
net.inet.carp.allow=1 
net.inet.carp.preempt=1 
net.inet.carp.log=2 

/etc/pf.conf 
pass quick on em0 inet proto carp from any to any 

ifconfig carp0 
carp0: flags=8843 mtu 1500 
        lladdr 00:00:5e:00:01:01 
        priority: 0 
        carp: carpdev em0 advbase 1 balancing ip-unicast 
                state BACKUP vhid 1 advskew 50 
                state MASTER vhid 2 advskew 0 
                state BACKUP vhid 3 advskew 100 
        groups: carp 
        status: backup 
        inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x4 
        inet 172.0.0.16 netmask 0xffff0000 broadcast 172.0.255.255 

NODE 3


/etc/hostname.em0 
inet 172.0.0.21 255.255.0.0 NONE 

/etc/hostname.carp0 
inet 172.0.0.16 255.255.0.0 172.0.255.255 balancing ip-unicast carpnodes 1:100,2:50,3:0 pass "password" carpdev em0 

/etc/sysctl.conf 
net.inet.carp.allow=1 
net.inet.carp.preempt=1 
net.inet.carp.log=2 

/etc/pf.conf 
pass quick on em0 inet proto carp from any to any 

ifconfig carp0 
carp0: flags=8843 mtu 1500 
        lladdr 00:00:5e:00:01:01 
        priority: 0 
        carp: carpdev em0 advbase 1 balancing ip-unicast 
                state BACKUP vhid 1 advskew 100 
                state BACKUP vhid 2 advskew 50 
                state MASTER vhid 3 advskew 0 
        groups: carp 
        status: backup 
        inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x4 
        inet 172.0.0.16 netmask 0xffff0000 broadcast 172.0.255.255 

On every node

Firewall disabled


tcpdump | grep CARP 
12:08:37.098390 CARPv2-advertise 36: vhid=3 advbase=1 advskew=50 demote=0 (DF) [tos 0x10] 
12:08:37.098533 CARPv2-advertise 36: vhid=3 advbase=1 advskew=50 demote=0 (DF) [tos 0x10] 
12:08:37.271021 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 
12:08:37.271028 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 
12:08:37.948303 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 
12:08:37.948415 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 
12:08:38.280916 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 
12:08:38.280921 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 
12:08:38.308264 CARPv2-advertise 36: vhid=3 advbase=1 advskew=50 demote=0 (DF) [tos 0x10] 
12:08:38.308395 CARPv2-advertise 36: vhid=3 advbase=1 advskew=50 demote=0 (DF) [tos 0x10] 
12:08:38.958201 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 
12:08:38.958305 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 
12:08:39.290790 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 
12:08:39.290808 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 (DF) [tos 0x10]

Thank you for your help.

networking
vmware-esxi
openbsd
carp
asked on Server Fault Jul 17, 2014 by Ouioui • edited Jul 17, 2014 by (unknown user)

2 Answers

1

Use the following filter on tcpdump to check the source address for CARP requests: tcpdump -pni em0 vrrp

After you enable net.reversePathFwsCheckPromisc flag, you have to disable and reenable the promiscuous mode to activate.

answered on Server Fault Jul 19, 2014 by denisvm
1

Had the same issue. The strange thing is that I have some ESXi servers on 1Gbit and some on 10Gbit linked to different switches. Whenever I move any OpenBSD (5.8) system on a a host with 10Gbit network got these errors in the logs. Following this article https://kb.vmware.com/s/article/59235 I did:

esxcli system settings advanced set -o /Net/ReversePathFwdCheckPromisc -i 1

Then move your interfaces off and back on promiscuous mode and worked just fine.


User contributions licensed under CC BY-SA 3.0