"ip rule to" works but "ip rule fwmark" fails - why?

I have a CentOS 6 (kernel 2.6.32) router with working OpenVPN client on it, and I want to redirect some traffic via VPN server.

Client (192.168.60.159) sends request to router (192.168.60.6:1443), and router redirects it via VPN connection (10.200.0.54) to server (185.61.149.21:443).

I've created a specific routing table tunde and a rule to use this table on packets MARKed by iptables. This rule has default GW on VPN while main routing table has real GW. When I try to use ip rule add fwmark 0x64 lookup tunde, it fails. Request passes to server OK, server replies, router receives reply but doesn't pass it to client.

But, if I add ip rule add to 185.61.149.21 lookup tunde - all works perfect. (But this rule doesn't satisfy my needs, i need per-port routing)

Looks like iptables somehow cannot de-masquerade replies.

Any ideas? Thank you!

#ip rule ls
32765:  from all fwmark 0x64 lookup tunde

This routing table has default gateway on VPN:

#ip route ls table tunde
10.200.0.53 dev tunde  proto kernel  scope link  src 10.200.0.54
192.168.60.0/24 dev eth0  proto kernel  scope link  src 192.168.60.6
default via 10.200.0.53 dev tunde  src 10.200.0.54

While main routing table has real default gateway:

# ip route ls
10.200.0.53 dev tunde  proto kernel  scope link  src 10.200.0.54
192.168.60.0/24 dev eth0  proto kernel  scope link  src 192.168.60.6
default via 192.168.60.1 dev eth0

Tcpdump shows both requests and replies on TUN interface:

# tcpdump -i tunde -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tunde, link-type RAW (Raw IP), capture size 65535 bytes
15:50:37.136708 IP 10.200.0.54.51409 > 185.61.149.21.443: Flags [S], seq 302961260, win 65280, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
15:50:37.209278 IP 185.61.149.21.443 > 10.200.0.54.51409: Flags [S.], seq 390829204, ack 302961261, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0
15:50:38.219458 IP 185.61.149.21.443 > 10.200.0.54.51409: Flags [S.], seq 390829204, ack 302961261, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0
15:50:40.136933 IP 10.200.0.54.51409 > 185.61.149.21.443: Flags [S], seq 302961260, win 65280, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
15:50:40.182989 IP 185.61.149.21.443 > 10.200.0.54.51409: Flags [S.], seq 390829204, ack 302961261, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0
15:50:42.191772 IP 185.61.149.21.443 > 10.200.0.54.51409: Flags [S.], seq 390829204, ack 302961261, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0
15:50:42.892051 IP 185.61.149.21.443 > 10.200.0.54.51391: Flags [S.], seq 528990609, ack 3345061728, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0

In iptables' log I also see replies. All packets are MARKed and use correct routing table:

# tail -f /var/log/messages
Oct 10 15:50:37 toy2 kernel: INPUT from Cli:  IN=eth0 OUT= MAC=00:15:5d:3c:bc:03:1c:39:47:f0:74:87:08:00 SRC=192.168.60.159 DST=192.168.60.6 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=19050 DF PROTO=TCP SPT=51409 DPT=1443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64
Oct 10 15:50:37 toy2 kernel: Forward To TUN:   IN=eth0 OUT=tunde SRC=192.168.60.159 DST=185.61.149.21 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=19050 DF PROTO=TCP SPT=51409 DPT=443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64
Oct 10 15:50:37 toy2 kernel: OUTPUT To TUN:  IN= OUT=tunde SRC=192.168.60.159 DST=185.61.149.21 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=19050 DF PROTO=TCP SPT=51409 DPT=443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64
Oct 10 15:50:37 toy2 kernel: INPUT from TUN:  IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51409 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64
Oct 10 15:50:38 toy2 kernel: INPUT from TUN:  IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51409 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64
Oct 10 15:50:40 toy2 kernel: INPUT from Cli:  IN=eth0 OUT= MAC=00:15:5d:3c:bc:03:1c:39:47:f0:74:87:08:00 SRC=192.168.60.159 DST=192.168.60.6 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=19063 DF PROTO=TCP SPT=51409 DPT=1443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64
Oct 10 15:50:40 toy2 kernel: Forward To TUN:   IN=eth0 OUT=tunde SRC=192.168.60.159 DST=185.61.149.21 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=19063 DF PROTO=TCP SPT=51409 DPT=443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64
Oct 10 15:50:40 toy2 kernel: INPUT from TUN:  IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51409 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64
Oct 10 15:50:42 toy2 kernel: INPUT from TUN:  IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51409 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64
Oct 10 15:50:42 toy2 kernel: INPUT from TUN:  IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51391 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64

(see MARK=0x64 on outgoing and incoming packets)

Now, iptables rules:

# iptables-save

*mangle
:PREROUTING ACCEPT [6278:3182515]
:INPUT ACCEPT [4169:3043489]
:FORWARD ACCEPT [9:468]
:OUTPUT ACCEPT [677:98865]
:POSTROUTING ACCEPT [703:101438]
-A PREROUTING -d 192.168.60.6/32 -i eth0 -p tcp -m tcp --dport 1443 -m state --state NEW -j CONNMARK --set-xmark 0x64/0xffffffff
-A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -m connmark --mark 0x64 -j MARK --set-xmark 0x64/0xffffffff
-A PREROUTING -m state --state NEW -m connmark ! --mark 0x0 -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -i tunde -j LOG --log-prefix " INPUT from TUN:  "
-A PREROUTING -s 192.168.60.159/32 -i eth0 -p tcp -m tcp --dport 1443 -j LOG --log-prefix " INPUT from Cli:  "
COMMIT

*nat
:PREROUTING ACCEPT [3487:307264]
:POSTROUTING ACCEPT [57:13668]
:OUTPUT ACCEPT [57:13668]
-A PREROUTING -d 192.168.60.6/32 -i eth0 -p tcp -m tcp --dport 1443 -j DNAT --to-destination 185.61.149.21:443
-A POSTROUTING -o tunde -j LOG --log-prefix " OUTPUT To TUN:  "
-A POSTROUTING -d 192.168.60.159/32 -o eth0 -p tcp -m tcp --sport 1443 -j LOG --log-prefix " OUTPUT To Cli:  "
-A POSTROUTING -o tun+ -j MASQUERADE
COMMIT

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [680:99605]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m multiport --dports 22,23 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 1443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -i tunde -j LOG --log-prefix " Forward From TUN: "
-A FORWARD -o tunde -j LOG --log-prefix " Forward To TUN:   "
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -i eth0 -o tun+ -j ACCEPT
-A FORWARD -i tun+ -o eth0 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

SOLVED!

With detailed logging I found out that packets were dropped after mangle/PREROUTING and not entering nat/PREROUTING. This is because of Kernel default handling of "unwanted" packets.

An

# echo 0 >/proc/sys/net/ipv4/conf/$interface/rp_filter

disables this filter mechanism and everything works fine again.