Transparent HAproxy on ubuntu (amazon EC2)

I am trying to setup a transparent HAproxy load balancer using ubuntu 18.04 on amazon EC2. I got the HAproxy to work on a non-transparent mode, so now I am trying to add the changes required for the transparent mode, but I am getting 503 Service Unavailable No server is available to handle this request. error.

An added issue is that the backend servers are located in another EC2 account. I mentioning it because some tutorial I have seen telling to change things in VPC configuration, subnets and routing tables. In my problem, the backend servers and the HAproxy are not in the same account, so definitely not in the same subnet/VPC.

I have disabled the "Actions > Network > Change source/Dest. check" for the load balacer instance in amazon EC2 dashboard.

TPROXY

# grep TPROXY /boot/config-4.15.0-10*
/boot/config-4.15.0-1021-aws:CONFIG_NETFILTER_XT_TARGET_TPROXY=m
/boot/config-4.15.0-1031-aws:CONFIG_NETFILTER_XT_TARGET_TPROXY=m

I needed to manually activate the xt_TPROXY on the kernel:

# sudo modprobe -v xt_TPROXY
insmod /lib/modules/4.15.0-1021-aws/kernel/net/netfilter/xt_TPROXY.ko

# lsmod | grep -i tproxy
xt_TPROXY              20480  0
nf_defrag_ipv6         36864  3 nf_conntrack_ipv6,xt_socket,xt_TPROXY
nf_defrag_ipv4         16384  3 xt_socket,nf_conntrack_ipv4,xt_TPROXY
x_tables               40960  10 iptable_mangle,ip_tables,iptable_filter,xt_mark,xt_socket,xt_TPROXY,xt_recent,ip6table_filter,xt_conntrack,ip6_tables

haproxy.cfg:

global
daemon
log /dev/log local4
maxconn 40000
ulimit-n 81000

defaults
log global
contimeout 4000
clitimeout 42000
srvtimeout 43000

listen http1
bind *:80 transparent
mode http
option http-server-close
option forwardfor
source 0.0.0.0 usesrc clientip
balance roundrobin
server http1_1 123.123.123.123:80 cookie http1_1 check inter 2000 rise 2 fall 3

Networking:

# cat /proc/sys/net/ipv4/conf/lo/rp_filter
0
# sysctl net.ipv4.ip_nonlocal_bind
net.ipv4.ip_nonlocal_bind = 1
# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

iptable mangle:

# iptables-save -t mangle
# Generated by iptables-save v1.6.1 on Tue Jan  1 15:35:54 2019
*mangle
:PREROUTING ACCEPT [216:12398]
:INPUT ACCEPT [30415:1434643]
:FORWARD ACCEPT [12:608]
:OUTPUT ACCEPT [57979:5590669]
:POSTROUTING ACCEPT [57991:5591277]
:DIVERT - [0:0]
-A PREROUTING -p tcp -m socket -j DIVERT
-A DIVERT -j MARK --set-xmark 0x6f/0xffffffff
-A DIVERT -j ACCEPT
COMMIT

HAproxy version (compiled with USE_LINUX_TPROXY):

# /usr/bin/haproxy -vv
HA-Proxy version 1.5-dev7 2011/09/10
Copyright 2000-2011 Willy Tarreau <w@1wt.eu>

Build options :
  TARGET  = linux26
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing
  OPTIONS = USE_LINUX_TPROXY=1 USE_STATIC_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Available polling systems :
     sepoll : pref=400,  test result OK
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 4 (4 usable), will use sepoll.

These are the results when running the HAproxy with the -d option. It is logging the incoming request, but there is no HTTP/1.1 200 OK reply from the backend server like on the non-transparent case:

# /usr/bin/haproxy -d -f /etc/haproxy.cfg
Available polling systems :
     sepoll : pref=400,  test result OK
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 4 (4 usable), will use sepoll.
Using sepoll() as the polling mechanism.
00000001:http1.accept(0004)=0006 from [111.222.333.111:54270]
00000002:http1.accept(0004)=0007 from [111.222.333.111:54269]
00000002:http1.clireq[0007:ffff]: GET /mytest.php HTTP/1.1
00000002:http1.clihdr[0007:ffff]: Host: example.com
00000002:http1.clihdr[0007:ffff]: Connection: keep-alive
00000002:http1.clihdr[0007:ffff]: Cache-Control: max-age=0
00000002:http1.clihdr[0007:ffff]: Upgrade-Insecure-Requests: 1
00000002:http1.clihdr[0007:ffff]: User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36
00000002:http1.clihdr[0007:ffff]: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
00000002:http1.clihdr[0007:ffff]: Accept-Encoding: gzip, deflate
00000002:http1.clihdr[0007:ffff]: Accept-Language: en-US,en;q=0.9,he;q=0.8,es;q=0.7
00000001:http1.clicls[0006:ffff]
00000001:http1.closed[0006:ffff]

ip rule list

0:      from all lookup local
32765:  from all fwmark 0x6f lookup 100
32766:  from all lookup main
32767:  from all lookup default

Any idea if what I am trying to do is possible? Do I miss anything? If you need me to add more info, let me know

iptables -L -t mangle

• (Fixed now) Don't do this unless you're looking for matching packets counters. It's absolutely not-human-friendly output compared to:

iptables-save -t mangle

• For transparent proxy there's TRPOXY target. Not sure if you're using it or not ("thanks" to iptables -L) but just in case: looks like you're missing it.

• (Fixed now) I don't find "transparent binding" in excerpts you've given. Check out the official manual on haproxy in transparent mode:

frontend ft_application
  bind 1.1.1.1:80 transparent

• You're using alternative routing table but do you have any routes installed inside of it?