HAProxy transparent, iproute2, iptables connmark


I'm trying to achieve high available transparent HAProxy setup.

For testing purposes I have the next setup (simplified sceme)

lb1 [] ----\
                        ---- backend1 []
lb2 [] ----/

Will not provide configuration on the lb{1,2}. It works totally fine. On the backend there is iptables rules to mark connections by mac-address:

-A PREROUTING -p tcp -m tcp --dport 8080 -m mac --mac-source <lb1's mac> -j MARK --set-xmark 0x1/0xffffffff
-A PREROUTING -p tcp -m tcp --dport 8080 -m mac --mac-source <lb2's mac> -j MARK --set-xmark 0x2/0xffffffff
-A PREROUTING -p tcp -m tcp --dport 8080 -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A OUTPUT -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

And additional routing tables:

0:  from all lookup local 
100:    from all fwmark 0x1 lookup lb1 
100:    from all fwmark 0x2 lookup lb2
32766:  from all lookup main 
32767:  from all lookup default
~]# ip r s table lb1
default via dev eth0

~]# ip r s table lb2
default via dev eth0

Everything works fine if there is default route in the main table and it uses the same device (eth0 in that case). But if there is no default route in the main table or default route uses different interface (lets say eth1) than default interface in custom tables the setup will not work.

I found that without default interface in the main table source IP-address becoming to address of the load balancing instead of the real client IP (external IP-address). Like:

output IN= OUT=eth0 SRC= DST= LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=8080 DPT=55209 WINDOW=14480 RES=0x00 ACK SYN URGP=0 MARK=0x2 

In that case I have to use different devices for default route in the main table and for default route in the custom tables. I suppose that there is some logic during the routing decision on the output path but can't find any documentation about it except source code which I'll investigate for that reason. May be someone know the reason and how to avoid it.

Thank you!

asked on Stack Overflow Feb 5, 2020 by choooze

0 Answers

Nobody has answered this question yet.

User contributions licensed under CC BY-SA 3.0