HAProxy Docker Container Doing L4 Load Balancing Not Being Transparent

1

Using the official latest HAProxy Docker container and the following config file:

frontend logging_frontend
    bind *:1514
    mode tcp
    timeout client 1m
    default_backend logging_backend

backend logging_backend
    mode tcp
    balance roundrobin
    timeout connect 10s
    timeout server 1m
    server logstash-collector-01 logstash-collector-01:1514 check
    server logstash-collector-02 logstash-collector-02:1514 check
    server logstash-collector-03 logstash-collector-03:1514 check

I'm getting logs (sent via nxlog to 1514 over TCP) which don't store the actual client IP but rather the Docker gateway IP. For example:

2017-03-02T15:57:41.585Z 172.18.0.1 {"EventTime":"2017-03-01 15:25:02","Hostname":"server.mycompany.net","Keywords":58096435992,"EventType":"ERROR","SeverityValue":4,"Severity":"ERROR","EventID":551,"SourceName":"Microsoft-Windows-SMBServer","ProviderGuid":"{D48CE617-33A2-4BC3-A5C7-11AA4F29619E}","Version":1,"Task":551,"OpcodeValue":0,"RecordNumber":5267740,"ProcessID":4,"ThreadID":5732,"Channel":"Microsoft-Windows-SMBServer/Security","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-184","AccountType":"User","Message":"SMB Session Authentication Failure\r\n\r\nClient Name: \\10.1.1.43\r\nClient Address: 10.1.1.43:54118\r\nUser Name: \r\nSession ID: 0xAF9FC8000015\r\nStatus: The attempted logon is invalid. This is either due to a bad username or authentication information. (0xC000006D)\r\n\r\nGuidance:\r\n\r\nYou should expect this error when attempting to connect to shares using incorrect credentials.\r\n\r\nThis error does not always indicate a problem with authorization, but mainly authentication. It is more common with non-Windows clients.\r\n\r\nThis error can occur when using incorrect usernames and passwords with NTLM, mismatched LmCompatibility settings between client and server, duplicate Kerberos service principal names, incorrect Kerberos ticket-granting service tickets, or Guest accounts without Guest access enabled","Opcode":"Info","EventReceivedTime":1488470262,"SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog"}

Note that the reporting server is 10.2.3.95, the Docker host is 10.1.38.223, and the Docker bridge interface (and local container subnet) is 172.18.0.0/16 (with 172.18.0.1, logged in the log, being the gateway).

From a tcpdump (modified to show interfaces): tcpdump

  1. The external interface, eno16777728, gets a PUSH packet (containing log data) from 10.2.3.95 to the host IP 10.1.38.223
  2. This gets DNAT'd by Docker to go from 10.2.3.95 to HAProxy container (172.18.0.6)
  3. HAProxy acknowledges the PUSH (final packet of screenshot shows the SNAT'd reply back on the external interface using 10.x.x.x IPs)
  4. (Out of order in image - check timestamps) HAProxy decides on backend container 172.18.0.3 for load balancing destination
  5. (Out of order in image - check timestamps) The Docker interface gateway SNATs the packet on its virtual ethernet interface (WHY?)
  6. ACK from backend server to Docker gateway
  7. ACK from backend server to HAProxy (?)
  8. ACK from backend server to HAProxy (?)

When I try to make HAProxy pass on the client IP transparently using source 0.0.0.0 usesrc clientip in the backend stanza, nothing gets logged at all.

To me, this question has two avenues to success; either:

  • How can the behavior of logging the Docker gateway IP be explained?

or more importantly:

  • How can I get the original client IP to appear in these logs, preferably as the client IP field (at beginning of log following timestamp)?
networking
load-balancing
haproxy
docker
containers
asked on Server Fault Mar 2, 2017 by armani

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0