netcat over a port of a secondary NIC

We are trying to establish communication between nc <public address1> and nc -l 30000 using a secondary NIC on ubuntu 14.04, called em1 on network 192.168.0.0/24.

Both NICS have independent internet access through independent gateways/routers.

To do that we've created a routing table, a couple rules to mark the packets and added the ip routes. See below.

As you can see, iptables marks packets related to port 30000, ip rule then tells the kernel to use the ftptable instead of the default one and this table has the 192.168.0.1/24 as default route.

I am not sure the routes are correct, given the results. And the packets seem to arrive because we have rule matches with but the messages do not go through.

If we listen with another computer on the same network it does work. If we do it from a machine in the network towards the server, it works (thanks to the 192.168.0.0/24 rule).

Is mangling active by default? What are we missing here?

# ip rule list
0:  from all lookup local 
32765:  from all fwmark 0x1 lookup ftptable 
32766:  from all lookup main 
32767:  from all lookup default 

.

# ip route show table ftptable
default via 192.168.0.1 dev em1 
192.168.0.0/24 dev em1  proto kernel  scope link  src 192.168.0.2 
192.168.30.0/24 dev p4p1  proto kernel  scope link  src 192.168.30.240 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 

.

# iptables-save
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*security
:INPUT ACCEPT [4040903:3466094909]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2985425:13178502885]
COMMIT
# Completed on Fri Sep 22 17:52:00 2017
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*raw
:PREROUTING ACCEPT [4235010:3593851556]
:OUTPUT ACCEPT [3083663:13237232624]
COMMIT
# Completed on Fri Sep 22 17:52:00 2017
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*nat
:PREROUTING ACCEPT [18035:2084634]
:INPUT ACCEPT [9322:747039]
:OUTPUT ACCEPT [7009:591525]
:POSTROUTING ACCEPT [7009:591525]
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
COMMIT
# Completed on Fri Sep 22 17:52:00 2017
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*mangle
:PREROUTING ACCEPT [7497:609073]
:INPUT ACCEPT [7342:587369]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [17006:47385884]
:POSTROUTING ACCEPT [17006:47385884]
-A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -m mark ! --mark 0x0 -j ACCEPT
-A PREROUTING -p tcp -m mark --mark 0x0 -m tcp --dport 30000 -j MARK --set-xmark 0x1/0xffffffff
-A INPUT -p tcp -m tcp --dport 30000 --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "EM1: "
-A INPUT -p tcp -m tcp --sport 30000 --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "EM1: "
-A INPUT -i em1 -p tcp -m tcp --dport 30000 -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -p tcp -m tcp --dport 30000 -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -p tcp -m tcp --dport 30000 --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "EM1: "
-A POSTROUTING -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
# Completed on Fri Sep 22 17:52:00 2017
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*filter
:INPUT ACCEPT [1173459:1591522133]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [599656:3734127129]
:fail2ban-proftpd - [0:0]
:fail2ban-ssh - [0:0]
-A INPUT -p tcp -m multiport --dports 21,20,990,989 -j fail2ban-proftpd
-A INPUT -p tcp -m multiport --dports 22 -j fail2ban-ssh
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 30000 --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "EM1 PACKET: "
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A fail2ban-proftpd -j RETURN
-A fail2ban-ssh -s 52.166.112.31/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-ssh -s 77.72.85.100/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-ssh -j RETURN
COMMIT
# Completed on Fri Sep 22 17:52:00 2017

Hit matches:

# iptables -vL -n -t mangle
Chain PREROUTING (policy ACCEPT 554 packets, 125K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 187M   51G CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x0
   11   660 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0 tcp dpt:30000 MARK set 0x1

Chain INPUT (policy ACCEPT 485 packets, 120K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   14   840 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30000 flags:0x17/0x02 LOG flags 0 level 4 prefix "EM1: "
    1    60 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:30000 flags:0x17/0x02 LOG flags 0 level 4 prefix "EM1: "
   23  1307 MARK       tcp  --  em1    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30000 MARK set 0x1

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 344 packets, 118K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    2   120 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30000 MARK set 0x1
    2   120 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30000 flags:0x17/0x02 LOG flags 0 level 4 prefix "EM1: "
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30000 MARK set 0x1

Chain POSTROUTING (policy ACCEPT 344 packets, 118K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 132M  635G CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save