Nginx IP transparency reverse proxying but upstreams need to be able to initiate connections to Internet

0

I have a docker swarm consisting of an Nginx service that serves as a reverse proxy for a couple of other upstream services, web and mail. Both web and mail service need to connect to an external service via HTTPS in order to perform authentication. The mail service needs to be able to connect to external servers via port 587 to send mail. Upstream web and mail need to be able to acquire the client's IP address so I have set up IP transparency on the Nginx server and the upstream web and mail servers as documented here https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/ (see Method 1). Inbound connections to web and mail work fine. They go through the Nginx proxy, then gets sent upstream to either web or mail and both see the actual client's IP address. However outbound connections initiated from inside web and mail don't make it through. Both web and mail have their default gateways set up to use the IP address of the Nginx. Normally this would be solved by adding SNAT rules or MASQUERADE using iptables. However if I do this, it breaks IP transparency and the upstream would be getting the IP address for the Nginx proxy instead of the client's IP address. Following the example in the document I linked to:

# ip rule add fwmark 1 lookup 100
# ip route add local 0.0.0.0/0 dev lo table 100
# iptables -t mangle -A PREROUTING -p tcp -s 172.16.0.0/28 --sport 80 -j MARK --set-xmark 0x1/0xffffffff

On web:

# route del default gw 172.16.0.1
# route add default gw 172.16.0.12

On mail:

# route del default gw 172.16.0.1
# route add default gw 172.16.0.13

Adding the IP masquerading rule (should be rewritten to SNAT in production) on the Nginx server:

# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# iptables -A FORWARD -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
# iptables -A FORWARD -i eth0 -j ACCEPT

I've tried filtering marked packets in the masquerade rule but it does not seem to work. For example:

# iptables -t nat -A POSTROUTING -o eth0 --match mark ! --mark 0x1/0xffffffff -j MASQUERADE

or

# iptables -t nat -A POSTROUTING -o eth0 --match mark --mark 0x1/0xffffffff -j MASQUERADE

I have tried varying combinations with POSTROUTING and PREROUTING but I can't get them to work.

Is there another way to get traffic out of the upstream servers and on to the Internet besides SNAT or IP masquerading? Or is there some way to set up filtering so that only unmarked packets are masqueraded?

EDIT: Just some clarification...

I already have IP transparency proxying working properly. I can connect to the Nginx proxy using the appropriate port, it forwards it to the upstream (either mail or web) and it would get the IP address of the client and not the IP address of the Nginx proxy and the client would get the response back from the upstream without any problems.

The problem is when the upstream servers need to initiate a connection to an external server. Such is the case for the mail server, which needs to connect to another server on the Internet to deliver mail. When the upstream server tries to do this the packets do not get out of the Docker network.

TL;DR: IP transparency in the Nginx proxy is working. But upstream servers cannot connect to the internet, because their default gateway has been set to the Nginx proxy's IP address.

linux
networking
nginx
iptables
docker
asked on Server Fault Aug 30, 2019 by n.abing • edited Sep 3, 2019 by n.abing

1 Answer

0

Do you want to make an email/web request and having your client's IP as source? If so this will not work, the response message will never get back to you but to your client's instead.

I believe you need to use Nginx headers for this to work and make your external web/mail servers aware of these headers.

In DNS resolvers this is called Client subnet option, where the client IP/prefix is bundled as a header in the DNS request message, while the source IP is the resolver.

answered on Server Fault Sep 2, 2019 by felartu

User contributions licensed under CC BY-SA 3.0