Routing subnet to specific routing table with fwmark, direct to ISP and VPN

2

Introduction

I'm trying to set up a VPN on my router so that I have two routing tables. I want it so that all the filtering is done with iptables, and iproute just acts on the specific fwmarks, ie generic rules.

It is an improvement of Routing a particular subnet into a VPN tunnel

The current working configuration of this full project can be found

http://wiki.alpinelinux.org/wiki/Linux_Router_with_VPN_on_a_Raspberry_Pi#VPN_Tunnel_on_specific_subnet

The main problem with that solution while it works and I am using it currently it requires filtering in both iptables and ip rules and therefore isn't very flexible.

The idea is that packets marked with 1 go directly out ppp0, and packets marked with 2 go through tun0.

Summary

  • The ISP table routes all traffic from 192.168.1.0/24 out ppp0
  • The VPN table routes all traffic from 192.168.2.0/24 out tun0

Network Diagram

http://i.stack.imgur.com/xQncJ.png

First I added both routing tables:

gateway:~# cat /etc/iproute2/rt_tables
1 ISP
2 VPN

Routing Scripts

Rules that are added when the ppp0 goes up on boot. Note I'm using the pppd hooks to keep things generic https://ppp.samba.org/pppd.html#sect13

gateway:~# cat /etc/ppp/ip-up 
#!/bin/sh
#
# This script is run by pppd when there's a successful ppp connection.
#

# Flush out any old routes when ppp0 goes down
/sbin/ip route flush table ISP

# Add a route for this subnet to the ISP table
/sbin/ip route add 192.168.1.0/24 dev eth0 table ISP prio 1

# Add a route from the ISP table
/sbin/ip rule add from ${IPLOCAL} table ISP prio 1

# Set default route to ppp0
/sbin/ip route add table ISP default via ${IPLOCAL} prio 1

Rules that are added when the VPN goes up. OpenVPN also has environmental variables: https://openvpn.net/index.php/open-source/documentation/manuals/65-openvpn-20x-manpage.html#lbAS

# cat /etc/openvpn/route-up-fwmark.sh 
#!/bin/sh
#
# This script is run by OpenVPN when there's a successful VPN connection.
#

# Flush out any old routes when ppp0 goes down
/sbin/ip route flush table VPN

# Add a route for this subnet to the VPN table
/sbin/ip route add 192.168.2.0/24 dev eth0 table VPN prio 2

# Add a route from the VPN table
/sbin/ip rule add from ${route_vpn_gateway} table VPN prio 2

# Set default route to tun0
/sbin/ip route add default via ${route_vpn_gateway} dev ${dev} table VPN prio 2

Routing Tables

main:

gateway:~# ip route sh table main
default dev ppp0  scope link  metric 300
172.16.32.0/20 dev tun0  proto kernel  scope link  src 172.16.39.64
192.168.0.0/30 dev eth1  proto kernel  scope link  src 192.168.0.2
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
192.168.2.0/24 dev eth0  proto kernel  scope link  src 192.168.2.1
${IPREMOTE} dev ppp0  proto kernel  scope link  src ${IPLOCAL}

ISP:

gateway:~# ip route sh table ISP
default via ${IPLOCAL} dev ppp0
192.168.1.0/24 dev eth0  scope link

VPN:

gateway:~# ip route sh table VPN
default via 172.16.32.1 dev tun0
192.168.2.0/24 dev eth0  scope link

IP Rules

In /etc/network/interfaces I added this under one of the interfaces:

post-up /etc/network/fwmark_2_0_subnet_rules

which contains:

gateway:~#  cat /etc/network/fwmark_2_0_subnet_rules
#!/bin/sh

/sbin/ip rule add fwmark 1 table ISP prio 1
/sbin/ip rule add fwmark 2 table VPN prio 2

Finally all the ip rules from above:

gateway:~# ip rule
0: from all lookup local
1: from all fwmark 0x1 lookup ISP
1: from <PPP IP ADDRESS> lookup ISP
2: from all fwmark 0x2 lookup VPN
2: from 172.16.32.1 lookup VPN
32766: from all lookup main
32767: from all lookup default

IPTables rules

#########################################################################
# Advanced routing rule set
# Uses 192.168.1.0 via ISP
#      192.168.2.0 via VPN
#
# Packets to/from 192.168.1.0/24 are marked with 0x1 and routed to ISP
# Packets to/from 192.168.2.0/24 are marked with 0x2 and routed to VPN
#
# http://nerdboys.com/2006/05/05/conning-the-mark-multiwan-connections-using-iptables-mark-connmark-and-iproute2/
# http://nerdboys.com/2006/05/08/multiwan-connections-addendum/
#########################################################################

# Set up the mangle table
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]

# Restore CONNMARK to the MARK (If one doesn't exist then no mark is set
-A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

# If packet MARK is 2, then it means there is already a connection mark and the original packet came in on VPN
-A PREROUTING -s 192.168.2.0/24 -m mark --mark 0x2 -j ACCEPT

# Else MARK packet as 2
#-A PREROUTING -i tun0 -j MARK --set-xmark 0x2/0xffffffff
-A PREROUTING -i tun0 -m conntrack --ctstate NEW -m mark --mark 0x0 -j MARK --set-xmark 0x2/0xffffffff

# If packet MARK is 1, then it means there is already a connection mark and the original packet came in on ISP
-A PREROUTING -s 192.168.1.0/24 -m mark --mark 0x1 -j ACCEPT

# Else MARK packet as 1
#-A PREROUTING -i ppp0 -j MARK --set-xmark 0x1/0xffffffff
-A PREROUTING -i ppp0 -m conntrack --ctstate NEW -m mark --mark 0x0 -j MARK --set-xmark 0x1/0xffffffff

# Save MARK to CONNMARK
-A PREROUTING -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT

# Set up the filter table
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]

# Create rule chain per input interface for forwarding packets
:FWD_ETH0 - [0:0]
:FWD_ETH1 - [0:0]
:FWD_PPP0 - [0:0]
:FWD_TUN0 - [0:0]

# Create rule chain per input interface for input packets (for host itself)
:IN_ETH0 - [0:0]
:IN_ETH1 - [0:0]
:IN_PPP0 - [0:0]
:IN_TUN0 - [0:0]

# Pass input packet to corresponded rule chain
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j IN_ETH0
-A INPUT -i eth1 -j IN_ETH1
-A INPUT -i ppp0 -j IN_PPP0
-A INPUT -i tun0 -j IN_TUN0

# TCP flag checks - block invalid flags
-A INPUT -m conntrack --ctstate INVALID -j DROP

# Log packets that are dropped in INPUT chain (useful for debugging)
-A INPUT -j LOG --log-prefix "iptables/filter/INPUT end"

# Pass forwarded packet to corresponded rule chain
-A FORWARD -i eth0 -j FWD_ETH0
-A FORWARD -i eth1 -j FWD_ETH1
-A FORWARD -i ppp0 -j FWD_PPP0
-A FORWARD -i tun0 -j FWD_TUN0

# Log packets that are dropped in FORWARD chain (useful for debugging)
-A FORWARD -j LOG --log-prefix "iptables/filter/FORWARD end"

# Forward traffic to LAN
-A FWD_ETH0 -s 192.168.1.0/24 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# Forward traffic to VPN
-A FWD_ETH0 -s 192.168.2.0/24 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# Forward SSH packets from network to modem
-A FWD_ETH1 -s 192.168.0.0/30 -d 192.168.1.0/24 -p tcp -m tcp --sport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A FWD_ETH1 -s 192.168.0.0/30 -d 192.168.2.0/24 -p tcp -m tcp --sport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# Forward traffic to ppp0 WAN port
-A FWD_PPP0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Forward ICMP from VPN, (breaks traceroute through VPN if you don't have this)
-A FWD_TUN0 -d 192.168.2.0/24 -p icmp -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Forward traffic to tun0 VPN port
-A FWD_TUN0 -d 192.168.2.0/24 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# SSH to Router
-A IN_ETH0 -s 192.168.1.0/24 -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A IN_ETH0 -s 192.168.2.0/24 -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# DNS to Router
-A IN_ETH0 -s 192.168.1.0/24 -p udp -m udp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
-A IN_ETH0 -s 192.168.2.0/24 -p udp -m udp --dport 53 -m conntrack --ctstate NEW -j ACCEPT

# FreeRadius Client
-A IN_ETH0 -s 192.168.1.0/24 -p tcp -m tcp --dport 1812 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A IN_ETH0 -s 192.168.1.0/24 -p udp -m udp --dport 1812 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# Ubiquiti UAP Device Discovery Broadcast
-A IN_ETH0 -s 192.168.1.0/24 -p udp -m udp --dport 10001 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# NTP
-A IN_ETH0 -s 192.168.1.0/24 -p udp -m udp --dport 123 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A IN_ETH0 -s 192.168.2.0/24 -p udp -m udp --dport 123 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# Accept traffic to router on both subnets
-A IN_ETH0 -s 192.168.1.0/24 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A IN_ETH0 -s 192.168.2.0/24 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# Prevent leakages from 192.168.2.0/24 hosts when VPN goes down for some reason
-A IN_ETH0 -s 192.168.2.0/24 -o ppp0 -j REJECT --reject-with icmp-port-unreachable

# SSH To Modem from Router
-A IN_ETH1 -s 192.168.0.0/30 -d 192.168.0.0/30 -p tcp -m tcp --sport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# Accept incoming tracked PPP0 connections
-A IN_PPP0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Incoming ICMP from VPN, (breaks traceroute through VPN if you don't have this)
-A IN_TUN0 -d 192.168.2.0/24 -p icmp -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Accept incoming tracked connections from 192.168.2.0/24 to VPN
-A IN_TUN0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]

# Bittorrent forwarded to Linux Workstation through VPN
-A PREROUTING -i tun0 -p tcp -m tcp --dport 6881:6889 -j DNAT --to-destination 192.168.2.20
-A PREROUTING -i tun0 -p udp -m udp --dport 6881:6889 -j DNAT --to-destination 192.168.2.20

# Allows for network hosts to access the internet via VPN tunnel
-A POSTROUTING -s 192.168.2.0/24 -o tun0 -j MASQUERADE

# Allows for network hosts to access the internet via WAN port
-A POSTROUTING -s 192.168.1.0/24 -o ppp0 -j MASQUERADE
COMMIT

I suspect the problem resides with my mangle table. It seems some of the packets are being marked:

gateway:~# iptables -L --line-numbers -n -v -t mangle
Chain PREROUTING (policy ACCEPT 1577 packets, 139K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     1577  139K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
2        0     0 ACCEPT     all  --  *      *       192.168.2.0/24       0.0.0.0/0            mark match 0x2
3        0     0 MARK       all  --  tun0   *       0.0.0.0/0            0.0.0.0/0            ctstate NEW mark match 0x0 MARK set 0x2
4        0     0 ACCEPT     all  --  *      *       192.168.1.0/24       0.0.0.0/0            mark match 0x1
5      112  6720 MARK       all  --  ppp0   *       0.0.0.0/0            0.0.0.0/0            ctstate NEW mark match 0x0 MARK set 0x1
6     1577  139K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save

Chain INPUT (policy ACCEPT 758 packets, 68909 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 819 packets, 69715 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 620 packets, 99208 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 1380 packets, 166K bytes)
num   pkts bytes target     prot opt in     out     source               destination

Any other improvements welcome, I plan on updating that AlpineLinux wiki article with the changes.

networking
router
routing
iptables
openvpn
asked on Super User Aug 3, 2015 by moonmoon • edited Aug 10, 2015 by moonmoon

1 Answer

0

I finally figured out a solution to this one.

First, as with the previous question edit and add to /etc/iproute2/rt_tables

1 ISP
2 VPN

Next up add to /etc/network/interfaces

auto eth0:2
iface eth0:2 inet static
    address 192.168.2.1
    netmask 255.255.255.0
    post-up /etc/network/fwmark_rules

Create the file /etc/network/fwmark_rules in here we out the fwmark rules:

#!/bin/sh

# Normal packets to go direct out WAN
/sbin/ip rule add fwmark 1 table ISP prio 100

# Put packets destined into VPN when VPN is up
/sbin/ip rule add fwmark 2 table VPN prio 200

# Prevent packets from being routed out when VPN is down.
# This prevents packets from falling back to the main table
# that has a priority of 32766
/sbin/ip rule add prohibit fwmark 2 prio 300

The prohibit line prevents things in table VPN from falling back to main if the VPN goes down as it's priority is less than 32766. You can read about prohibit and other special rules in 4.9. Routing Policy Database (RPDB)

Add /etc/ppp/ip-up. This is where we put the ip rules for when the PPP connection comes online.

#!/bin/sh
#
# This script is run by pppd when there's a successful ppp connection.
#

# Flush out any old rules that might be there
/sbin/ip route flush table ISP

# Add route to table from subnets on LAN
/sbin/ip route add 192.168.1.0/24 dev eth0 table ISP prio 100
/sbin/ip route add 192.168.2.0/24 dev eth0 table ISP prio 200

# Add route from IP given by ISP to the table
/sbin/ip rule add from ${IPLOCAL} table ISP prio 100

# Add a default route
/sbin/ip route add table ISP default via ${IPLOCAL} prio 100

Make sure to add /etc/ppp/ip-down this is where we delete the rule when the interface comes down so we don't get duplicates. You can read about the special hook values in pppd man file - Scripts

#!/bin/sh
#
# This script is run by pppd after the connection has ended.
#

# Delete the rules when we take the interface down
/sbin/ip rule del from ${IPLOCAL} table ISP prio 100

Now we need to create the routing scripts for OpenVPN. So edit /etc/openvpn/route-up-fwmark.sh OpenVPN also has special environmental variables OpenVPN - Environmental Variables)

#!/bin/sh
#
# This script is run by OpenVPN when there's a successful VPN connection.
#

# Flush out any old rules that might be there
/sbin/ip route flush table VPN

# Add route to table from 192.168.2.0/24 subnet on LAN
/sbin/ip route add 192.168.2.0/24 dev eth0 table VPN prio 200

# Add route from VPN interface IP to the VPN table
/sbin/ip rule add from ${ifconfig_local} table VPN prio 200

# Add a default route
/sbin/ip route add default via ${ifconfig_local} dev ${dev} table VPN prio 200

And to delete the rule just before the interface shuts down /etc/openvpn/route-pre-down-fwmark.sh

#!/bin/sh
#
# This script is run by OpenVPN after the connection has ended
#

# Delete the rules when we take the interface down
/sbin/ip rule del from ${ifconfig_local} table VPN prio 200

Finally your mangle table should look like this:

*mangle

# Set default policies for table
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]

# Restore CONNMARK to the MARK (If one doesn't exist then no mark is set)
-A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

# If packet MARK is 2, then it means there is already a connection mark and the original packet came in on VPN
-A PREROUTING -s 192.168.2.0/24 -m mark --mark 0x2 -j ACCEPT

# Check exception (this is a server which when accessed on a 192.168.2.0/24 address will go out the ISP table) are 0x1
#-A PREROUTING -s 192.168.2.0/24 -d <IP_OF_EXCEPTED_SERVER>/32 -m mark --mark 0x1 -j ACCEPT

# Check packets coming from 192.168.2.0/24 are 0x2
-A PREROUTING -s 192.168.2.0/24 -j MARK --set-xmark 0x2/0xffffffff

# If packet MARK is 1, then it means there is already a connection mark and the original packet came in on ISP
-A PREROUTING -s 192.168.1.0/24 -m mark --mark 0x1 -j ACCEPT

# Check packets coming from 192.168.1.0/24 are 0x1
-A PREROUTING -s 192.168.1.0/24 -j MARK --set-xmark 0x1/0xffffffff

# Mark exception (this is a server which when accessed on a 192.168.2.0/24 address will go out the ISP table) as 0x1
#-A PREROUTING -s 192.168.2.0/24 -d <IP_OF_EXCEPTED_SERVER>/32 -j MARK --set-xmark 0x1/0xffffffff

# Save MARK to CONNMARK (remember iproute can't see CONNMARKs)
-A PREROUTING -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT

You will also want to edit /etc/openvpn/openvpn.conf and add

# Prevents default gateway from being set on the default routing table
route-noexec

# Allows route-up script to be executed
script-security 2

# Calls custom shell script after connection to add necessary routes
route-up /etc/openvpn/route-up-fwmark.sh
route-pre-down /etc/openvpn/route-pre-down-fwmark.sh
answered on Super User Aug 12, 2015 by moonmoon

User contributions licensed under CC BY-SA 3.0