Linux port based routing : reponse packets discarded

I would like to use port-based routing on my Linux local gateway. Here is my network diagram :

             192.168.42.148/24     192.168.42.1/24         192.168.44.2     192.168.44.1 
+--------------------+                +----------------------------+            +----------+
| Workstation (eth0)-|----------------|-(em0)   Local_GW    (tun0)-|------------|  VPN_GW  |
+--------------------+                |           (rl0)            |            +----------+
                                      +-------------|--------------+
                                10.133.8.79/21      |
                                                    |
                                10.133.15.254/21    |
                                        +--------------------+
                                        |     Provider_GW    |
                                        +--------------------+

Provider_GW only allows 21,110,143,554,587,993,995,5222,6666:6669 as outgoing ports for packets bound for the Internet. Local_GW is connected to VPN_GW through a UDP SSL VPN (OpenVPN)

Therefore, I would like to use :

• 10.133.15.254 as gateway for tcp packets whose outgoing port is in the list 21,110,143,554,587,993,995,5222,6666:6669
• 192.168.44.1 as gateway for tcp packets with any other outgoing port

Here are the steps I follow :

#!/bin/bash

## Flush FIP (Forbidden Internet Ports) table
ip route flush table FIP

## Copy all routes from main table except the default one
ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table FIP $ROUTE; done

## Add default route
ip route add default via 192.168.44.1 table FIP

## Flush iptables PREROUTING chain in table mangle
iptables -t mangle -F PREROUTING

## Create iptables rules for packet marking
iptables -t mangle -N MFIP
### Only mark packets bound for the Internet
iptables -t mangle -A MFIP --dst 192.168.0.0/16 --jump RETURN
iptables -t mangle -A MFIP --dst 172.16.0.0/12 --jump RETURN
iptables -t mangle -A MFIP --dst 10.0.0.0/8 --jump RETURN
iptables -t mangle -A MFIP --dst 169.254.0.0/16 --jump RETURN

iptables -t mangle -A MFIP --jump MARK --set-mark 1

iptables -t mangle -A PREROUTING -i em0 -p tcp -m multiport ! --dports 21,110,143,554,587,993,995,5222,6666:6669 --jump MFIP

## Delete old routing rule if it exist
ip rule del fwmark 1

## Create new routing rule
ip rule add fwmark 1 table FIP

## Zero Counters (useful for debugging)
iptables -t mangle -Z

---

I can't connect to an SSH server from Workstation :

fabien@Workstation:~$ ssh xion345@sdf.org
ssh: connect to host sdf.org port 22: Connection timed out

However I can see that packets are marked and routed as expected :
(On Local_GW) :

fabien@Local_GW:~$ sudo tcpdump -fi em0 tcp port 22 and host not 192.168.42.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:07:32.065071 IP LoopbackMarvin.local.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:35.064024 IP LoopbackMarvin.local.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:41.060423 IP LoopbackMarvin.local.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]

Local_GW indeed receives the SYN packets from workstation

fabien@Local_GW:~$ sudo iptables -t mangle -vL -n
Chain PREROUTING (policy ACCEPT 7046 packets, 829K bytes)
pkts bytes target     prot opt in     out     source               destination         
558 43499 MFIP       tcp  --  em0    *       0.0.0.0/0            0.0.0.0/0           multiport dports ! 21,110,143,554,587,993,995,5222,6666:6669 

Chain INPUT (policy ACCEPT 5690 packets, 713K bytes)
pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 220 packets, 49131 bytes)
pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 540 packets, 352K bytes)
pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 767 packets, 402K bytes)
pkts bytes target     prot opt in     out     source               destination         

Chain MFIP (1 references)
pkts bytes target     prot opt in     out     source               destination         
555 43319 RETURN     all  --  *      *       0.0.0.0/0            192.168.0.0/16      
    0     0 RETURN     all  --  *      *       0.0.0.0/0            172.16.0.0/12       
    0     0 RETURN     all  --  *      *       0.0.0.0/0            10.0.0.0/8          
    0     0 RETURN     all  --  *      *       0.0.0.0/0            169.254.0.0/16      
    3   180 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x1/0xffffffff 

Apparently, SYN packets from Workstation are correctly marked by iptables.

fabien@Local_GW:~$ sudo tcpdump -fi tun0 tcp port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
13:07:32.065153 IP 192.168.42.148.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:32.487217 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:35.064062 IP 192.168.42.148.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:35.341717 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:35.510967 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:41.060459 IP 192.168.42.148.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:41.336589 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:41.411675 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:53.411081 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]

The SYN packet from Workstation is correctly routed to VPN_GW and it even receives and answer (from the sdf.org ssh server): 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855. However, Local_GW doesn't route this response packet back to Workstation, Why ? I must be missing something obvious but I can't understand what.

Thank you very much for reading.

---

If you want the exact details regarding my configuration, here are the (very) verbose outputs from iptables and ip. Sorry if it makes my question very long, I try to be as precise as possible.

Firewall:

fabien@Local_GW:~$ sudo iptables -t nat -vL -n
Chain PREROUTING (policy ACCEPT 145K packets, 10M bytes)
pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 1934 packets, 137K bytes)
pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 1938 packets, 137K bytes)
pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      rl0     192.168.44.0/24      0.0.0.0/0           
668 43570 MASQUERADE  all  --  *      rl0     192.168.42.0/24      0.0.0.0/0      

fabien@Local_GW:~$ sudo iptables  -vL -n
Chain INPUT (policy ACCEPT 132K packets, 15M bytes)
pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 34345 packets, 24M bytes)
pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 15490 packets, 7741K bytes)
pkts bytes target     prot opt in     out     source               destination       

Routing tables:

fabien@Local_GW:~$ sudo ip route show
192.168.46.1 dev tun1  proto kernel  scope link  src 192.168.46.3 
192.168.44.1 dev tun0  proto kernel  scope link  src 192.168.44.2 
192.168.46.0/24 via 192.168.46.1 dev tun1 
192.168.42.0/24 dev em0  proto kernel  scope link  src 192.168.42.1 
192.168.43.0/24 via 192.168.44.1 dev tun0 
10.133.8.0/21 dev rl0  proto kernel  scope link  src 10.133.8.79 
169.254.0.0/16 dev em0  scope link  metric 1000 
default via 10.133.15.254 dev rl0  metric 100 

fabien@Local_GW:~$ sudo ip route show table FIP
192.168.44.1 dev tun0  proto kernel  scope link  src 192.168.44.2 
192.168.46.1 dev tun1  proto kernel  scope link  src 192.168.46.3 
192.168.46.0/24 via 192.168.46.1 dev tun1 
192.168.42.0/24 dev em0  proto kernel  scope link  src 192.168.42.1 
192.168.43.0/24 via 192.168.44.1 dev tun0 
10.133.8.0/21 dev rl0  proto kernel  scope link  src 10.133.8.79 
169.254.0.0/16 dev em0  scope link  metric 1000 
default via 192.168.44.1 dev tun0 

fabien@Zaphod:~$ ip rule show
0:      from all lookup local 
32765:  from all fwmark 0x1 lookup FIP 
32766:  from all lookup main
32767:  from all lookup default

Disabling the reverse-path filter solve this issue.

root@Local_GW:/# echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

However I can't understand why it is necessary. Is someone able to give more details about this issue ? 192.168.44.1 is set as default route in table FIP, the kernel should know that packets coming from the Internet may pass through tun0.

On the other hand, could disabling the reverse-path filter create security holes ?