Routing between OpenSWAN / IPSEC tunnels

1

I am trying to connect multiple Amazon VPCs (across regions) together using OpenSWAN and Amazon VGW's. The router instance can ping to hosts in both VPCs, and traffic is attempting to cross the router, but is getting dropped.

EDIT: I see the counter XfrmInNoPols incrementing when the pings are not being forwarded.

In this scenario there are two VPCs being connected, and a instance that happens to be in a 3rd VPC is performing the routing and acting as a hub. I am attempting to essentially reimplement the Transit VPC function (https://aws.amazon.com/blogs/aws/aws-solution-transit-vpc/), without the Cisco CSR and automated lambda configuration.

My issue is that the hub is able to reach both East and West, but packets from either end arrive on hub but reach no further.

Topology:

West (172.19.0.0/16) - (hub) - East (172.18.0.0/16). Hub is connecting to both ends via VGW's, so cleartext packets for east/west never leave Hub. As per normal VGW behavior, two tunnels exist between each end and HUB.

The basis for this configuration is https://github.com/patrickbcullen/Openswan-VPC, modified to support a 2nd set of tunnels. One oddity about this script is it set ups a 'network namespace' (http://man7.org/linux/man-pages/man8/ip-netns.8.html) to handle all the ipsec and routing.

The hub can ping nodes in east and west via the IPSEC tunnels. The VGW's agree that ipsec and BGP is up, the the East/West subnets see the propagated routes. The hub has routes to both East and West. Iptables is fully open. rp_filter is set to 0 and forwarding / ip_forward is set to 1 in sysctl.

I set up a ping generator in West that is attempting to ping East. The packets reach the openswan network namespace in hub:

16:38:49.311665 IP 35.163.220.45 > 169.254.255.3: ESP(spi=0x0a790d98,seq=0x4f5), length 132
16:38:49.311665 IP 172.19.58.64 > 172.18.57.207: ICMP echo request, id 411, seq 1113, length 64

I have NFLOG / ulogd2 setup in iptables. It shows:

RAW-PREROUTING IN=eth0 OUT= MAC=d6:fd:61:4b:73:42:6a:3a:bb:e2:33:75:08:00 SRC=172.19.58.64 DST=172.18.57.207 LEN=84 TOS=00 PREC=0x00 TTL=254 ID=49803 DF PROTO=ICMP TYPE=8 CODE=0 ID=411 SEQ=1155 MARK=0
NAT-PREROUTING IN=eth0 OUT= MAC=d6:fd:61:4b:73:42:6a:3a:bb:e2:33:75:08:00 SRC=172.19.58.64 DST=172.18.57.207 LEN=84 TOS=00 PREC=0x00 TTL=254 ID=49803 DF PROTO=ICMP TYPE=8 CODE=0 ID=411 SEQ=1155 MARK=0

However the packet never reaches the FORWARD iptables chain:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

Pinging from East to West fails similarly.

The hub can ping both the source and destination:

# ping -c 1 172.18.57.207
64 bytes from 172.18.57.207: icmp_seq=1 ttl=254 time=1.74 ms
# ping -c 1 172.19.58.64
64 bytes from 172.19.58.64: icmp_seq=1 ttl=254 time=94.3 ms

Any suggestions on what might be blocking packets from transiting hub?


The host is a AWS EC2 AMI, latest version:

Linux version 4.4.30-32.54.amzn1.x86_64 (mockbuild@gobi-build-60008) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov 10 15:52:05 UTC 2016ux
Linux Openswan U2.6.37/K4.4.30-32.54.amzn1.x86_64 (netkey)

My iptables rules (all ACCEPT, only NFLOGs):

# Generated by iptables-save v1.4.18 on Fri Nov 18 16:40:41 2016
*mangle
:PREROUTING ACCEPT [3648:404080]
:INPUT ACCEPT [2490:306808]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1802:171212]
:POSTROUTING ACCEPT [1802:171212]
-A INPUT -j NFLOG --nflog-prefix  MAN-INPUT --nflog-group 5
-A OUTPUT -j NFLOG --nflog-prefix  MAN-OUTPUT --nflog-group 5
-A POSTROUTING -j NFLOG --nflog-prefix  MAN-POSTROUTING --nflog-group 5
COMMIT
# Completed on Fri Nov 18 16:40:41 2016
# Generated by iptables-save v1.4.18 on Fri Nov 18 16:40:41 2016
*filter
:INPUT ACCEPT [2490:306808]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1802:171212]
-A INPUT -j NFLOG --nflog-prefix  FLT-INPUT --nflog-group 5
-A OUTPUT -j NFLOG --nflog-prefix  FLT-OUTPUT --nflog-group 5
COMMIT
# Completed on Fri Nov 18 16:40:41 2016
# Generated by iptables-save v1.4.18 on Fri Nov 18 16:40:41 2016
*raw
:PREROUTING ACCEPT [3648:404080]
:OUTPUT ACCEPT [1802:171212]
-A PREROUTING -j NFLOG --nflog-prefix  RAW-PREROUTING --nflog-group 5
-A OUTPUT -j NFLOG --nflog-prefix  RAW-OUTPUT --nflog-group 5
COMMIT
# Completed on Fri Nov 18 16:40:41 2016
# Generated by iptables-save v1.4.18 on Fri Nov 18 16:40:41 2016
*nat
:PREROUTING ACCEPT [1158:97272]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -j NFLOG --nflog-prefix  NAT-PREROUTING --nflog-group 5
-A POSTROUTING -j NFLOG --nflog-prefix  NAT-POSTROUTING --nflog-group 5
COMMIT

IPSec config:

# /etc/ipsec.conf - Openswan IPsec configuration file
#
# Manual:     ipsec.conf.5
#
# Please place your own config files in /etc/ipsec.d/ ending in .conf

version 2.0     # conforms to second version of ipsec.conf specification

# basic configuration
config setup
        protostack=netkey
        nat_traversal=no
        virtual_private=
        oe=off

conn awstunnel1
        authby=secret
        auto=start
        left=169.254.255.2
        leftid=169.254.255.2
        right=35.163.197.247
        rightid=35.163.197.247
        type=tunnel
        ikelifetime=8h
        keylife=1h
        phase2alg=aes128-sha1;modp1024
        ike=aes128-sha1
        auth=esp
        keyingtries=%forever
        aggrmode=no
        keyexchange=ike
        ikev2=never
        leftsubnet=169.254.12.53/30
        rightsubnet=0.0.0.0/0
        dpddelay=10
        dpdtimeout=30
        dpdaction=restart_by_peer
conn awstunnel2
        authby=secret
        auto=start
        left=169.254.255.3
        leftid=169.254.255.3
        right=35.163.220.45
        rightid=35.163.220.45
        type=tunnel
        ikelifetime=8h
        keylife=1h
        phase2alg=aes128-sha1;modp1024
        ike=aes128-sha1
        auth=esp
        keyingtries=%forever
        aggrmode=no
        keyexchange=ike
        ikev2=never
        leftsubnet=169.254.12.221/30
        rightsubnet=0.0.0.0/0
        dpddelay=10
        dpdtimeout=30
        dpdaction=restart_by_peer
conn awstunnel3
        authby=secret
        auto=start
        left=169.254.255.4
        leftid=169.254.255.4
        right=52.45.134.147
        rightid=52.45.134.147
        type=tunnel
        ikelifetime=8h
        keylife=1h
        phase2alg=aes128-sha1;modp1024
        ike=aes128-sha1
        auth=esp
        keyingtries=%forever
        aggrmode=no
        keyexchange=ike
        ikev2=never
        leftsubnet=169.254.47.13/30
        rightsubnet=0.0.0.0/0
        dpddelay=10
        dpdtimeout=30
        dpdaction=restart_by_peer
conn awstunnel4
        authby=secret
        auto=start
        left=169.254.255.5
        leftid=169.254.255.5
        right=52.45.232.151
        rightid=52.45.232.151
        type=tunnel
        ikelifetime=8h
        keylife=1h
        phase2alg=aes128-sha1;modp1024
        ike=aes128-sha1
        auth=esp
        keyingtries=%forever
        aggrmode=no
        keyexchange=ike
        ikev2=never
        leftsubnet=169.254.47.1/30
        rightsubnet=0.0.0.0/0
        dpddelay=10
        dpdtimeout=30
        dpdaction=restart_by_peer

(I'm omitting the secrets)

BGP configuration:

# cat /etc/quagga/bgpd.conf
hostname ip-172-28-10-214
password xx
enable password xx
!
log file /var/log/quagga/bgpd.log
debug bgp events
debug bgp zebra
debug bgp updates
debug bgp filters
debug bgp fsm
!
router bgp 65001
    bgp router-id 52.55.78.109
    network 169.254.12.54/30
    neighbor 169.254.12.53 remote-as 7224
    neighbor 169.254.12.53 soft-reconfiguration inbound
    neighbor 169.254.12.53 route-map rm_peer_1_out out
    network 169.254.12.222/30
    neighbor 169.254.12.221 remote-as 7224
    neighbor 169.254.12.221 soft-reconfiguration inbound
    neighbor 169.254.12.221 route-map rm_peer_1_out out
    network 169.254.47.14/30
    neighbor 169.254.47.13 remote-as 7224
    neighbor 169.254.47.13 soft-reconfiguration inbound
    neighbor 169.254.47.13 route-map rm_peer_1_out out
    network 169.254.47.2/30
    neighbor 169.254.47.1 remote-as 7224
    neighbor 169.254.47.1 soft-reconfiguration inbound
    neighbor 169.254.47.1 route-map rm_peer_1_out out
line vty
!
ip prefix-list localprefix seq 5 permit 172.18.0.0/16
ip prefix-list remoteprefix seq 5 permit any
! Suppress the AWS AS
route-map rm_peer_1_out permit 5
 match ip address prefix-list localprefix
 set as-path exclude 7224
! Suppress the AWS AS, synthetically extend the AS PATH
! For any vpc that isn't in the same region
route-map rm_peer_1_out permit 6
 match ip address prefix-list remoteprefix
 set as-path prepend 65001
 set as-path exclude 7224
! Suppress advertisement for non-VPC addresses
access-list vpcprefixes permit 172.0.0.0/8
!

Route table:

default via 169.254.255.1 dev eth0
169.254.12.52/30 dev eth0  proto kernel  scope link  src 169.254.12.54
169.254.12.220/30 dev eth0  proto kernel  scope link  src 169.254.12.222
169.254.47.0/30 dev eth0  proto kernel  scope link  src 169.254.47.2
169.254.47.12/30 dev eth0  proto kernel  scope link  src 169.254.47.14
169.254.255.0/28 dev eth0  proto kernel  scope link  src 169.254.255.2
172.18.0.0/16 via 169.254.47.13 dev eth0  proto zebra  metric 100
172.19.0.0/16 via 169.254.12.221 dev eth0  proto zebra  metric 100

sysctl:

net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.veth1.forwarding = 1
net.ipv4.ip_forward = 1

ipsec auto --status:

000 using kernel interface: netkey
000 interface lo/lo ::1
000 interface lo/lo 127.0.0.1
000 interface eth0/eth0 169.254.255.2
000 interface eth0/eth0 169.254.12.54
000 interface eth0/eth0 169.254.12.222
000 interface eth0/eth0 169.254.47.14
000 interface eth0/eth0 169.254.47.2
000 interface eth0/eth0 169.254.255.3
000 interface eth0/eth0 169.254.255.4
000 interface eth0/eth0 169.254.255.5
000 %myid = (none)
000 debug none
000
000 virtual_private (%priv):
000 - allowed 0 subnets:
000 - disallowed 0 subnets:
000 WARNING: Either virtual_private= is not specified, or there is a syntax
000          error in that line. 'left/rightsubnet=vhost:%priv' will not work!
000 WARNING: Disallowed subnets in virtual_private= is empty. If you have
000          private address space in internal use, it should be excluded!
000
[SNIP algorithms]
000
000 stats db_ops: {curr_cnt, total_cnt, maxsz} :context={0,8,64} trans={0,8,3072} attrs={0,8,2048}
000
000 "awstunnel1": 169.254.12.52/30===169.254.255.2<169.254.255.2>[+S=C]...35.163.197.247<35.163.197.247>[+S=C]===0.0.0.0/0; erouted; eroute owner: #8
000 "awstunnel1":     myip=unset; hisip=unset;
000 "awstunnel1":   ike_life: 28800s; ipsec_life: 3600s; rekey_margin: 540s; rekey_fuzz: 100%; keyingtries: 0
000 "awstunnel1":   policy: PSK+ENCRYPT+TUNNEL+PFS+UP+SAREFTRACK+lKOD+rKOD; prio: 30,0; interface: eth0;
000 "awstunnel1":   dpd: action:restart_by_peer; delay:10; timeout:30;
000 "awstunnel1":   newest ISAKMP SA: #1; newest IPsec SA: #8;
000 "awstunnel1":   IKE algorithms wanted: AES_CBC(7)_128-SHA1(2)_000-MODP1536(5), AES_CBC(7)_128-SHA1(2)_000-MODP1024(2); flags=-strict
000 "awstunnel1":   IKE algorithms found:  AES_CBC(7)_128-SHA1(2)_160-MODP1536(5), AES_CBC(7)_128-SHA1(2)_160-MODP1024(2)
000 "awstunnel1":   IKE algorithm newest: AES_CBC_128-SHA1-MODP1024
000 "awstunnel1":   ESP algorithms wanted: AES(12)_128-SHA1(2)_000; pfsgroup=MODP1024(2); flags=-strict
000 "awstunnel1":   ESP algorithms loaded: AES(12)_128-SHA1(2)_160
000 "awstunnel1":   ESP algorithm newest: AES_128-HMAC_SHA1; pfsgroup=MODP1024
000 "awstunnel2": 169.254.12.220/30===169.254.255.3<169.254.255.3>[+S=C]...35.163.220.45<35.163.220.45>[+S=C]===0.0.0.0/0; erouted; eroute owner: #7
000 "awstunnel2":     myip=unset; hisip=unset;
000 "awstunnel2":   ike_life: 28800s; ipsec_life: 3600s; rekey_margin: 540s; rekey_fuzz: 100%; keyingtries: 0
000 "awstunnel2":   policy: PSK+ENCRYPT+TUNNEL+PFS+UP+SAREFTRACK+lKOD+rKOD; prio: 30,0; interface: eth0;
000 "awstunnel2":   dpd: action:restart_by_peer; delay:10; timeout:30;
000 "awstunnel2":   newest ISAKMP SA: #2; newest IPsec SA: #7;
000 "awstunnel2":   IKE algorithms wanted: AES_CBC(7)_128-SHA1(2)_000-MODP1536(5), AES_CBC(7)_128-SHA1(2)_000-MODP1024(2); flags=-strict
000 "awstunnel2":   IKE algorithms found:  AES_CBC(7)_128-SHA1(2)_160-MODP1536(5), AES_CBC(7)_128-SHA1(2)_160-MODP1024(2)
000 "awstunnel2":   IKE algorithm newest: AES_CBC_128-SHA1-MODP1024
000 "awstunnel2":   ESP algorithms wanted: AES(12)_128-SHA1(2)_000; pfsgroup=MODP1024(2); flags=-strict
000 "awstunnel2":   ESP algorithms loaded: AES(12)_128-SHA1(2)_160
000 "awstunnel2":   ESP algorithm newest: AES_128-HMAC_SHA1; pfsgroup=MODP1024
000 "awstunnel3": 169.254.47.12/30===169.254.255.4<169.254.255.4>[+S=C]...52.45.134.147<52.45.134.147>[+S=C]===0.0.0.0/0; erouted; eroute owner: #5
000 "awstunnel3":     myip=unset; hisip=unset;
000 "awstunnel3":   ike_life: 28800s; ipsec_life: 3600s; rekey_margin: 540s; rekey_fuzz: 100%; keyingtries: 0
000 "awstunnel3":   policy: PSK+ENCRYPT+TUNNEL+PFS+UP+SAREFTRACK+lKOD+rKOD; prio: 30,0; interface: eth0;
000 "awstunnel3":   dpd: action:restart_by_peer; delay:10; timeout:30;
000 "awstunnel3":   newest ISAKMP SA: #3; newest IPsec SA: #5;
000 "awstunnel3":   IKE algorithms wanted: AES_CBC(7)_128-SHA1(2)_000-MODP1536(5), AES_CBC(7)_128-SHA1(2)_000-MODP1024(2); flags=-strict
000 "awstunnel3":   IKE algorithms found:  AES_CBC(7)_128-SHA1(2)_160-MODP1536(5), AES_CBC(7)_128-SHA1(2)_160-MODP1024(2)
000 "awstunnel3":   IKE algorithm newest: AES_CBC_128-SHA1-MODP1024
000 "awstunnel3":   ESP algorithms wanted: AES(12)_128-SHA1(2)_000; pfsgroup=MODP1024(2); flags=-strict
000 "awstunnel3":   ESP algorithms loaded: AES(12)_128-SHA1(2)_160
000 "awstunnel3":   ESP algorithm newest: AES_128-HMAC_SHA1; pfsgroup=MODP1024
000 "awstunnel4": 169.254.47.0/30===169.254.255.5<169.254.255.5>[+S=C]...52.45.232.151<52.45.232.151>[+S=C]===0.0.0.0/0; erouted; eroute owner: #6
000 "awstunnel4":     myip=unset; hisip=unset;
000 "awstunnel4":   ike_life: 28800s; ipsec_life: 3600s; rekey_margin: 540s; rekey_fuzz: 100%; keyingtries: 0
000 "awstunnel4":   policy: PSK+ENCRYPT+TUNNEL+PFS+UP+SAREFTRACK+lKOD+rKOD; prio: 30,0; interface: eth0;
000 "awstunnel4":   dpd: action:restart_by_peer; delay:10; timeout:30;
000 "awstunnel4":   newest ISAKMP SA: #4; newest IPsec SA: #6;
000 "awstunnel4":   IKE algorithms wanted: AES_CBC(7)_128-SHA1(2)_000-MODP1536(5), AES_CBC(7)_128-SHA1(2)_000-MODP1024(2); flags=-strict
000 "awstunnel4":   IKE algorithms found:  AES_CBC(7)_128-SHA1(2)_160-MODP1536(5), AES_CBC(7)_128-SHA1(2)_160-MODP1024(2)
000 "awstunnel4":   IKE algorithm newest: AES_CBC_128-SHA1-MODP1024
000 "awstunnel4":   ESP algorithms wanted: AES(12)_128-SHA1(2)_000; pfsgroup=MODP1024(2); flags=-strict
000 "awstunnel4":   ESP algorithms loaded: AES(12)_128-SHA1(2)_160
000 "awstunnel4":   ESP algorithm newest: AES_128-HMAC_SHA1; pfsgroup=MODP1024
000
000 #8: "awstunnel1":500 STATE_QUICK_I2 (sent QI2, IPsec SA established); EVENT_SA_REPLACE in 881s; newest IPSEC; eroute owner; isakmp#1; idle; import:admin initiate
000 #8: "awstunnel1" esp.5759bbc6@35.163.197.247 esp.7db002d9@169.254.255.2 tun.0@35.163.197.247 tun.0@169.254.255.2 ref=0 refhim=4294901761
000 #1: "awstunnel1":500 STATE_MAIN_I4 (ISAKMP SA established); EVENT_SA_REPLACE in 26389s; newest ISAKMP; lastdpd=-1s(seq in:0 out:0); idle; import:admin initiate
000 #7: "awstunnel2":500 STATE_QUICK_I2 (sent QI2, IPsec SA established); EVENT_SA_REPLACE in 1114s; newest IPSEC; eroute owner; isakmp#2; idle; import:admin initiate
000 #7: "awstunnel2" esp.c817fa78@35.163.220.45 esp.a790d98@169.254.255.3 tun.0@35.163.220.45 tun.0@169.254.255.3 ref=0 refhim=4294901761
000 #2: "awstunnel2":500 STATE_MAIN_I4 (ISAKMP SA established); EVENT_SA_REPLACE in 26003s; newest ISAKMP; lastdpd=-1s(seq in:0 out:0); idle; import:admin initiate
000 #5: "awstunnel3":500 STATE_QUICK_I2 (sent QI2, IPsec SA established); EVENT_SA_REPLACE in 1083s; newest IPSEC; eroute owner; isakmp#3; idle; import:admin initiate
000 #5: "awstunnel3" esp.98c8c16a@52.45.134.147 esp.70f458c4@169.254.255.4 tun.0@52.45.134.147 tun.0@169.254.255.4 ref=0 refhim=4294901761
000 #3: "awstunnel3":500 STATE_MAIN_I4 (ISAKMP SA established); EVENT_SA_REPLACE in 26042s; newest ISAKMP; lastdpd=-1s(seq in:0 out:0); idle; import:admin initiate
000 #6: "awstunnel4":500 STATE_QUICK_I2 (sent QI2, IPsec SA established); EVENT_SA_REPLACE in 864s; newest IPSEC; eroute owner; isakmp#4; idle; import:admin initiate
000 #6: "awstunnel4" esp.7f07c4fa@52.45.232.151 esp.80005db1@169.254.255.5 tun.0@52.45.232.151 tun.0@169.254.255.5 ref=0 refhim=4294901761
000 #4: "awstunnel4":500 STATE_MAIN_I4 (ISAKMP SA established); EVENT_SA_REPLACE in 26073s; newest ISAKMP; lastdpd=-1s(seq in:0 out:0); idle; import:admin initiate
000

BGP state:

# vtysh -c 'show ip bgp summary'
BGP router identifier 52.55.78.109, local AS number 65001
RIB entries 11, using 1056 bytes of memory
Peers 4, using 18 KiB of memory

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
169.254.12.53   4  7224     185     188        0    0    0 00:30:21        1
169.254.12.221  4  7224     185     187        0    0    0 00:30:23        1
169.254.47.1    4  7224     185     188        0    0    0 00:30:22        1
169.254.47.13   4  7224     185     187        0    0    0 00:30:22        1

# vtysh -c 'show ip bgp'
BGP table version is 0, local router ID is 52.55.78.109
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 169.254.12.52/30 0.0.0.0                  0         32768 i
*> 169.254.12.220/30
                    0.0.0.0                  0         32768 i
*> 169.254.47.0/30  0.0.0.0                  0         32768 i
*> 169.254.47.12/30 0.0.0.0                  0         32768 i
*> 172.18.0.0       169.254.47.13          100             0 7224 i
*                   169.254.47.1           200             0 7224 i
*  172.19.0.0       169.254.12.53          200             0 7224 i
*>                  169.254.12.221         100             0 7224 i

ip xfrm state (keys snipped) inside the namespace:

# ip xfrm state
src 35.163.197.247 dst 169.254.255.2
    proto esp spi 0x7db002d9 reqid 16385 mode tunnel
    replay-window 32 flag af-unspec
    auth-trunc hmac(sha1) x 96
    enc cbc(aes) x
    anti-replay context: seq 0xfc, oseq 0x0, bitmap 0xffffffff
src 169.254.255.2 dst 35.163.197.247
    proto esp spi 0x5759bbc6 reqid 16385 mode tunnel
    replay-window 32 flag af-unspec
    auth-trunc hmac(sha1) x 96
    enc cbc(aes) x
    anti-replay context: seq 0x0, oseq 0x180, bitmap 0x00000000
src 35.163.220.45 dst 169.254.255.3
    proto esp spi 0x0a790d98 reqid 16389 mode tunnel
    replay-window 32 flag af-unspec
    auth-trunc hmac(sha1) x 96
    enc cbc(aes) x
    anti-replay context: seq 0x8c0, oseq 0x0, bitmap 0xffffffff
src 169.254.255.3 dst 35.163.220.45
    proto esp spi 0xc817fa78 reqid 16389 mode tunnel
    replay-window 32 flag af-unspec
    auth-trunc hmac(sha1) x 96
    enc cbc(aes) x
    anti-replay context: seq 0x0, oseq 0x14b, bitmap 0x00000000
src 52.45.232.151 dst 169.254.255.5
    proto esp spi 0x80005db1 reqid 16397 mode tunnel
    replay-window 32 flag af-unspec
    auth-trunc hmac(sha1) x 96
    enc cbc(aes) x
    anti-replay context: seq 0xe9, oseq 0x0, bitmap 0xffffffff
src 169.254.255.5 dst 52.45.232.151
    proto esp spi 0x7f07c4fa reqid 16397 mode tunnel
    replay-window 32 flag af-unspec
    auth-trunc hmac(sha1) x 96
    enc cbc(aes) x
    anti-replay context: seq 0x0, oseq 0x180, bitmap 0x00000000
src 52.45.134.147 dst 169.254.255.4
    proto esp spi 0x70f458c4 reqid 16393 mode tunnel
    replay-window 32 flag af-unspec
    auth-trunc hmac(sha1) x 96
    enc cbc(aes) x
    anti-replay context: seq 0xfc, oseq 0x0, bitmap 0xffffffff
src 169.254.255.4 dst 52.45.134.147
    proto esp spi 0x98c8c16a reqid 16393 mode tunnel
    replay-window 32 flag af-unspec
    auth-trunc hmac(sha1) x 96
    enc cbc(aes) x
    anti-replay context: seq 0x0, oseq 0x17f, bitmap 0x00000000

ip xfrm policy inside the namespace:

# ip xfrm policy
src 169.254.12.52/30 dst 0.0.0.0/0
    dir out priority 2176 ptype main
    tmpl src 169.254.255.2 dst 35.163.197.247
        proto esp reqid 16385 mode tunnel
src 0.0.0.0/0 dst 169.254.12.52/30
    dir fwd priority 2176 ptype main
    tmpl src 35.163.197.247 dst 169.254.255.2
        proto esp reqid 16385 mode tunnel
src 0.0.0.0/0 dst 169.254.12.52/30
    dir in priority 2176 ptype main
    tmpl src 35.163.197.247 dst 169.254.255.2
        proto esp reqid 16385 mode tunnel
src 169.254.12.220/30 dst 0.0.0.0/0
    dir out priority 2176 ptype main
    tmpl src 169.254.255.3 dst 35.163.220.45
        proto esp reqid 16389 mode tunnel
src 0.0.0.0/0 dst 169.254.12.220/30
    dir fwd priority 2176 ptype main
    tmpl src 35.163.220.45 dst 169.254.255.3
        proto esp reqid 16389 mode tunnel
src 0.0.0.0/0 dst 169.254.12.220/30
    dir in priority 2176 ptype main
    tmpl src 35.163.220.45 dst 169.254.255.3
        proto esp reqid 16389 mode tunnel
src 169.254.47.0/30 dst 0.0.0.0/0
    dir out priority 2176 ptype main
    tmpl src 169.254.255.5 dst 52.45.232.151
        proto esp reqid 16397 mode tunnel
src 0.0.0.0/0 dst 169.254.47.0/30
    dir fwd priority 2176 ptype main
    tmpl src 52.45.232.151 dst 169.254.255.5
        proto esp reqid 16397 mode tunnel
src 0.0.0.0/0 dst 169.254.47.0/30
    dir in priority 2176 ptype main
    tmpl src 52.45.232.151 dst 169.254.255.5
        proto esp reqid 16397 mode tunnel
src 169.254.47.12/30 dst 0.0.0.0/0
    dir out priority 2176 ptype main
    tmpl src 169.254.255.4 dst 52.45.134.147
        proto esp reqid 16393 mode tunnel
src 0.0.0.0/0 dst 169.254.47.12/30
    dir fwd priority 2176 ptype main
    tmpl src 52.45.134.147 dst 169.254.255.4
        proto esp reqid 16393 mode tunnel
src 0.0.0.0/0 dst 169.254.47.12/30
    dir in priority 2176 ptype main
    tmpl src 52.45.134.147 dst 169.254.255.4
        proto esp reqid 16393 mode tunnel
src ::/0 dst ::/0
    socket out priority 0 ptype main
src ::/0 dst ::/0
    socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
    socket out priority 0 ptype main
[repeats snipped]

ip rule list inside the namespace:

# ip rule list
0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

ip addr list inside the namespace:

# ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
6: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:fd:61:4b:73:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.255.2/28 scope global eth0
       valid_lft forever preferred_lft forever
    inet 169.254.12.54/30 scope global eth0
       valid_lft forever preferred_lft forever
    inet 169.254.12.222/30 scope global eth0
       valid_lft forever preferred_lft forever
    inet 169.254.47.14/30 scope global eth0
       valid_lft forever preferred_lft forever
    inet 169.254.47.2/30 scope global eth0
       valid_lft forever preferred_lft forever
    inet 169.254.255.3/28 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet 169.254.255.4/28 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet 169.254.255.5/28 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::d4fd:61ff:fe4b:7342/64 scope link
       valid_lft forever preferred_lft forever
8: veth1@if7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8e:9a:f6:27:83:fe brd ff:ff:ff:ff:ff:ff link-netnsid 0

ifconfig inside the namespace:

# ifconfig -a
eth0      Link encap:Ethernet  HWaddr D6:FD:61:4B:73:42
          inet addr:169.254.255.2  Bcast:0.0.0.0  Mask:255.255.255.240
          inet6 addr: fe80::d4fd:61ff:fe4b:7342/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3803 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2076 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:570566 (557.1 KiB)  TX bytes:270108 (263.7 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

veth1     Link encap:Ethernet  HWaddr 8E:9A:F6:27:83:FE
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Pfkey:

# cat /proc/net/pfkey
sk       RefCnt Rmem   Wmem   User   Inode

Kernel config:

CONFIG_XFRM=y
CONFIG_XFRM_ALGO=m
CONFIG_XFRM_USER=m
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_STATISTICS=y
CONFIG_XFRM_IPCOMP=m
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_SECURITY_NETWORK_XFRM=y
amazon-web-services
vpn
linux-networking
ipsec
openswan
asked on Server Fault Nov 18, 2016 by Jason Martin • edited Nov 23, 2016 by Jason Martin

1 Answer

0

From what i see overall, your routing seems to be fine. and the fact that you can ping from one way to the other and the fact the ipsec tunnel is up tells me there is clearly a network communication that is established between the routers. SO we can move away from the layer 3 part of the problem

The key in situations like this is to Identify the problem first then Isolate it. The best way to find it would be to sniff the network traffic from the source of the last responsive hop to the destination. What type of traffic goes through the tunnel ?? Any chance you might have Jumbo frames ( MTU higher than 1500) ?? Is there any storage traffic like ISCSI or FcOE ?? It is the most common cause of traffic getting dropped on unix based VPN`s. Some drivers do not support MTU higher than 1500.

If that is the case and the drivers support it, increase the MTU of the TUN interfaces to 9000 on all end (clients and servers) look at the trace you sniffed, you are looking at 2 things. If jumbo frames are good to go and you get a lot of timeouts, it can be a tunnel based problem or software based problem. At that point, you will have to determine where the packets are dropped. To eliminate tunnel problems, it can be be dropped at the egress or ingress of any device, at any point of the flow. traceroute (from within the tunnel) is extremely valuable, otherwise you will have to sniff source and destination peer to peer until you identify where the drop is. If you receive a lot of TCP Reset, it is software based, so it is a layer 1 problem, and i can`t help you with that since i am not a programmer ;)

answered on Server Fault Nov 23, 2016 by Greenlord • edited Nov 23, 2016 by Greenlord

User contributions licensed under CC BY-SA 3.0