I am trying use a port from two applications and have each of them receive the packet from a different set of IP addresses. In order to achieve this, I use the SO_REUSEPORT and SO_ATTACH_REUSEPORT_CBPF socket options. My code is as follows:
parentfd = socket(AF_INET, SOCK_STREAM, 0);
if (parentfd < 0)
error( "ERROR opening socket");
struct sock_filter code[]={
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 3, 0x00000800 },
{ 0x20, 0, 0, 0x0000001a },
{ 0x15, 2, 0, 0xc0a8ff01 },
{ 0x6, 0, 0, 0x00000000 },
{ 0x6, 0, 0, 0x00040000 },
{ 0x6, 0, 0, 0x00000001 },
};
struct sock_fprog bpf = {
.len = ARRAY_SIZE(code),
.filter = code,
};
if (setsockopt(parentfd, SOL_SOCKET, SO_REUSEPORT, (const void *)&optval,sizeof(optval)))
error("ERROR setting SO_REUSEPORT");
if (setsockopt(parentfd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, (const void *)&bpf, sizeof(bpf)))
error("ERROR setting SO_ATTACH_REUSEPORT_CBPF);
I also have a different process that listens to the same port using only the SO_REUSEPORT flag. From a machine with IP 192.168.255.1
I am running echo 1234 | ncat 192.168.255.150 1234
. Based on my filter I would expect all traffic from that IP address to be received by the second process. However, it is all received by the first. When I change the filter to a simple:
struct sock_filter code[]={
{ 0x6, 0, 0, 0x00000001 },
};
It works as expected and all packets are received by the second process. Any idea why this might be happening?
I found out what the problem was. The filter is applied to all packets, even the TCP handshake packets. Also, the base pointer points to the first byte of the packet payload, not headers. Hence, when it executes
ldh[12]
it gets out of the limits of the packet (SYN packet has 0 bytes of payload) and the default behavior is to return 0.
The non-working code is:
l0: ldh [12] /* read EtherType (2 bytes), which is found at offset 12 (decimal) */
l1: jeq #0x800, l2, l5 /* if EtherType == `0x800` (IPv4), jump to `l2`, otherwise jump to `l5` */
l2: ld [26] /* read source IP address (4 bytes) */
l3: jeq #0xc0a8ff01, l6, l4 /* if source IP address == 192.168.255.1, jump to l6 (return 1), else jump to l4 (return 0) */
l4: ret #0
l5: ret #0x40000
l6: ret #0x1
The working code is:
ret #0x1
socket (7)
says:
The BPF program must return an index between 0 and N-1 representing the socket which should receive the packet (where N is the number of sockets in the group). If the BPF program returns an invalid index, socket selection will fall back to the plain SO_REUSEPORT mechanism.
On my machine tcpdump -i lo -ddd 'src host 192.168.255.1'
produces
10
40 0 0 12
21 0 2 2048
32 0 0 26
21 4 5 3232300801
21 1 0 2054
21 0 3 32821
32 0 0 28
21 0 1 3232300801
6 0 0 262144
6 0 0 0
Which is
l0: ldh [12]
l1: jeq #0x800, l2, l4
l2: ld [26]
l3: jeq #0xc0a8ff01, l8, l9
l4: jeq #0x806, l6, l5
l5: jeq #0x8035, l6, l9
l6: ld [28]
l7: jeq #0xc0a8ff01, l8, l9
l8: ret #0x40000
l9: ret #0
I fail to see anything that's obviously wrong with your code.
Have you tried running tcpdump on the server? Perhaps you've forgotten to remove an extra IP address on the client, or there's a forgotten SNAT rule somewhere?
Which kernel version are you running? Could you post a minimal C application that reproduces the problem?
User contributions licensed under CC BY-SA 3.0