Route ftp packets to one network interface, the rest to the other

I have an ubuntu 14.04 server with two NICs, each one connected to seldom routers, each with separated internet access.

We want to redirect ftp ports traffic through the second interface, em1, and all the rest through the default interface p4p1.

I have both interfaces up. I've followed the instructions on the first option of this question

To sum up, I've created a table, marked the packets and added ip routes.

But ftp to the public address of em1 times out. (the router forwards tcp/udp traffic on ftp port to the server's em1) Also, the public address of p4p1 still responds normally to ftp requests.

which is the proper way to achieve this?

bonus: I'd be nice if the first interface p4p1 also is able to handle ftp requests, but the priority is that the bulk of traffic goes through em1.

EDIT:

Until I figure this out for the ftp ports, I'm trying with a high port, 30000 and netcat. I have a nc -l 30000 and I'm trying to connect with another computer with nc <em1 public> 30000. I've tried many mangle markings

~# iptables -vL -t mangle
Chain PREROUTING (policy ACCEPT 70M packets, 21G bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp  --  em1    any     anywhere             anywhere             tcp spt:30000 MARK set 0x1

Chain INPUT (policy ACCEPT 70M packets, 21G bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 44M packets, 244G bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp  --  any    any     anywhere             anywhere             tcp spt:30000 MARK set 0x1
    0     0 MARK       tcp  --  any    any     anywhere             anywhere             tcp dpt:30000 MARK set 0x1

Chain POSTROUTING (policy ACCEPT 44M packets, 244G bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CHECKSUM   udp  --  any    virbr0  anywhere             anywhere             udp dpt:bootpc CHECKSUM fill

.

$# ip rule list
0:  from all lookup local 
32764:  from all fwmark 0x1 lookup ftptable 
32765:  from all fwmark 0x1 lookup ftptable 
32766:  from all lookup main 
32767:  from all lookup default

.

$# ip route show table ftptable
default via 192.168.0.1 dev em1 
192.168.0.0/24 dev em1  proto kernel  scope link  src 192.168.0.2 
192.168.30.0/24 dev p4p1  proto kernel  scope link  src 192.168.30.240 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 

netcat connects when I use the private addresses, both p4p1 and em1, probably because they are on that routing table. But if I use the router public address, it doesn't connect (netcat doesn't say anything).

Also, if I set the router forward to another computer with only one interface connected to the em1 network, it works, so the router is redirecting the packets correctly.

Some packets are matching, what am I missing?

$# iptables -vL
Chain INPUT (policy ACCEPT 110K packets, 18M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 6665  350K fail2ban-proftpd  tcp  --  any    any     anywhere             anywhere             multiport dports ftp,ftp-data,ftps,ftps-data
32902 3536K fail2ban-ssh  tcp  --  any    any     anywhere             anywhere             multiport dports ssh
    0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere             udp dpt:domain
    0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere             tcp dpt:domain
    0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere             udp dpt:bootps
    0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere             tcp dpt:bootps
    2   120 LOG        tcp  --  any    any     anywhere             anywhere             tcp dpt:30000 flags:FIN,SYN,RST,ACK/SYN LOG level warning prefix "EM1 PACKET: "

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  any    virbr0  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  virbr0 any     192.168.122.0/24     anywhere            
    0     0 ACCEPT     all  --  virbr0 virbr0  anywhere             anywhere            
    0     0 REJECT     all  --  any    virbr0  anywhere             anywhere             reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr0 any     anywhere             anywhere             reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT 51159 packets, 415M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  any    virbr0  anywhere             anywhere             udp dpt:bootpc

Chain fail2ban-proftpd (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 6065  320K RETURN     all  --  any    any     anywhere             anywhere            

Chain fail2ban-ssh (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REJECT     all  --  any    any     52.166.112.31        anywhere             reject-with icmp-port-unreachable
    3   180 REJECT     all  --  any    any     77.72.85.100         anywhere             reject-with icmp-port-unreachable
31246 3423K RETURN     all  --  any    any     anywhere             anywhere            

.

$# iptables -vL -t mangle
Chain PREROUTING (policy ACCEPT 84103 packets, 11M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp  --  em1    any     anywhere             anywhere             tcp spt:30000 MARK set 0x1

Chain INPUT (policy ACCEPT 82011 packets, 11M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp  --  em1    any     anywhere             anywhere             tcp spt:30000 MARK set 0x1
    0     0 MARK       tcp  --  em1    any     anywhere             anywhere             tcp dpt:30000 MARK set 0x1

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 29709 packets, 405M bytes)
 pkts bytes target     prot opt in     out     source               destination         
   14   760 MARK       tcp  --  any    any     anywhere             anywhere             tcp spt:30000 MARK set 0x1
    6   336 MARK       tcp  --  any    any     anywhere             anywhere             tcp dpt:30000 MARK set 0x1

Chain POSTROUTING (policy ACCEPT 29716 packets, 405M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CHECKSUM   udp  --  any    virbr0  anywhere             anywhere             udp dpt:bootpc CHECKSUM fill

EDIT: Output of iptables-save after adding the rules suggested in the answer. I've also added logging rules for debugging.

# iptables-save
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*security
:INPUT ACCEPT [4040903:3466094909]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2985425:13178502885]
COMMIT
# Completed on Fri Sep 22 17:52:00 2017
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*raw
:PREROUTING ACCEPT [4235010:3593851556]
:OUTPUT ACCEPT [3083663:13237232624]
COMMIT
# Completed on Fri Sep 22 17:52:00 2017
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*nat
:PREROUTING ACCEPT [18035:2084634]
:INPUT ACCEPT [9322:747039]
:OUTPUT ACCEPT [7009:591525]
:POSTROUTING ACCEPT [7009:591525]
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
COMMIT
# Completed on Fri Sep 22 17:52:00 2017
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*mangle
:PREROUTING ACCEPT [7497:609073]
:INPUT ACCEPT [7342:587369]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [17006:47385884]
:POSTROUTING ACCEPT [17006:47385884]
-A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -m mark ! --mark 0x0 -j ACCEPT
-A PREROUTING -p tcp -m mark --mark 0x0 -m tcp --dport 30000 -j MARK --set-xmark 0x1/0xffffffff
-A INPUT -p tcp -m tcp --dport 30000 --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "EM1: "
-A INPUT -p tcp -m tcp --sport 30000 --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "EM1: "
-A INPUT -i em1 -p tcp -m tcp --dport 30000 -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -p tcp -m tcp --dport 30000 -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -p tcp -m tcp --dport 30000 --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "EM1: "
-A POSTROUTING -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
# Completed on Fri Sep 22 17:52:00 2017
# Generated by iptables-save v1.4.21 on Fri Sep 22 17:52:00 2017
*filter
:INPUT ACCEPT [1173459:1591522133]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [599656:3734127129]
:fail2ban-proftpd - [0:0]
:fail2ban-ssh - [0:0]
-A INPUT -p tcp -m multiport --dports 21,20,990,989 -j fail2ban-proftpd
-A INPUT -p tcp -m multiport --dports 22 -j fail2ban-ssh
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 30000 --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "EM1 PACKET: "
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A fail2ban-proftpd -j RETURN
-A fail2ban-ssh -s 52.166.112.31/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-ssh -s 77.72.85.100/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-ssh -j RETURN
COMMIT
# Completed on Fri Sep 22 17:52:00 2017

I've also changed sysctl values, as I've seen other posts suggesting it:

net.ipv4.conf.default.rp_filter=2
net.ipv4.conf.all.rp_filter=2
net.ipv4.ip_forward=1
sysctl -w net.ipv4.conf.em1.rp_filter=2

Update the routing table ftptable and add a gateway for your default route. Currently, once your ftp packets are switched to using the ftptable, it doesn't know how to go back out the public IP's gateway.

For policy route I usually use a combination of:

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

And then for ftp specifically:

iptables -t mangle -A PREROUTING -m mark --mark 0 -p tcp --dport 21 -j MARK --set-mark 1

Some variation may be required if this is capturing too much FTP traffic instead of only incoming. Related connections inherit the parent mark, so they do not need a specific rule.

If it's still not working, the output of iptables-save is a bit more precise than iptables -vL and could be helpful for analysis.

Another useful tool for diagnostics is conntrack. You can use conntrack -L to dump the table and view marks.