I have two hosts, one is Charity with IP 67.207.128.184, the other is Hope with IP 67.207.130.204. I've installed the ipsec-tools package on each host. I have not installed racoon on either host. Charity has IPTables configured, whereas Hope does not. The following is a report on how IPTables is configured on each host (with irrelevant sensitive information removed):
charity# iptables -L -v
Chain INPUT (policy ACCEPT 63 packets, 6072 bytes)
 pkts bytes target     prot opt in     out     source               destination 
   49  3160 ACCEPT     all  --  lo     any     anywhere             anywhere    
 6012  989K ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
  208 12536 ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:www
   13   824 ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:https
    6   364 ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:ssh
    0     0 ACCEPT     esp  --  any    any     anywhere             anywhere    
    0     0 ACCEPT     ah   --  any    any     anywhere             anywhere    
    3   245 ACCEPT     icmp --  any    any     anywhere             anywhere            icmp echo-request
   31  1759 LOG        all  --  any    any     anywhere             anywhere            LOG level debug prefix `iptables debug: '
   31  1759 REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-port-unreachable
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination 
    0     0 ACCEPT     esp  --  any    any     anywhere             anywhere    
    0     0 ACCEPT     ah   --  any    any     anywhere             anywhere    
    0     0 REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT 47 packets, 4920 bytes)
 pkts bytes target     prot opt in     out     source               destination 
 5121 3650K ACCEPT     all  --  any    any     anywhere             anywhere    
hope# iptables -L -v
Chain INPUT (policy ACCEPT 1194 packets, 195K bytes)
 pkts bytes target     prot opt in     out     source               destination 
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination 
Chain OUTPUT (policy ACCEPT 1035 packets, 452K bytes)
 pkts bytes target     prot opt in     out     source               destination 
I have configured Charity (67.207.128.184) with an /etc/ipsec-tools.conf file like so:
#!/usr/sbin/setkey -f
# Flush the SAD and SPD
flush;
spdflush;
# AH SAs using 128 bit long keys
add 67.207.128.184 67.207.130.204 ah 0x200 -A hmac-md5
    0x18caf1d5d58297471cd63a21fd02adb6;
add 67.207.130.204 67.207.128.184 ah 0x300 -A hmac-md5
    0xde8ba4d945acbe6e85db0978b0c30184;
# ESP SAs using 192 bit long keys (168 + 24 parity)
add 67.207.128.184 67.207.130.204 esp 0x201 -E 3des-cbc
    0xa3f3baeaff7ad1cc3c00df7ad4b2bd26e4af4a703308431b;
add 67.207.130.204 67.207.128.184 esp 0x301 -E 3des-cbc
    0xf51e36bb6640072612366e134b5919f60f5a0af22fefa7fb;
# Security policies
spdadd 67.207.128.184 67.207.130.204 any -P out ipsec
    esp/transport//require
    ah/transport//require;
spdadd 67.207.130.204 67.207.128.184 any -P in ipsec
    esp/transport//require
    ah/transport//require;
I've configured Hope (67.207.130.204) with an /etc/ipsec-tools.conf file like so:
#!/usr/sbin/setkey -f
# Flush the SAD and SPD
flush;
spdflush;
# AH SAs using 128 bit long keys
add 67.207.128.184 67.207.130.204 ah 0x200 -A hmac-md5
    0x18caf1d5d58297471cd63a21fd02adb6;
add 67.207.130.204 67.207.128.184 ah 0x300 -A hmac-md5
    0xde8ba4d945acbe6e85db0978b0c30184;
# ESP SAs using 192 bit long keys (168 + 24 parity)
add 67.207.128.184 67.207.130.204 esp 0x201 -E 3des-cbc
    0xa3f3baeaff7ad1cc3c00df7ad4b2bd26e4af4a703308431b;
add 67.207.130.204 67.207.128.184 esp 0x301 -E 3des-cbc
    0xf51e36bb6640072612366e134b5919f60f5a0af22fefa7fb;
# Security policies
spdadd 67.207.128.184 67.207.130.204 any -P in ipsec
    esp/transport//require
    ah/transport//require;
spdadd 67.207.130.204 67.207.128.184 any -P out ipsec
    esp/transport//require
    ah/transport//require;
I've restarted /etc/init.d/setkey after making changes to these files. I'm running tcpdump on Hope, and I SSH from Charity to Hope and see the following in the tcpdump logs:
18:46:11.218238 IP charity.progclub.org > hope: AH(spi=0x00000200,seq=0x40): ESP(spi=0x00000201,seq=0x40), length 64    
18:46:11.218361 IP hope > charity.progclub.org: AH(spi=0x00000300,seq=0x22): ESP(spi=0x00000301,seq=0x22), length 64    
18:46:11.218822 IP charity.progclub.org > hope: AH(spi=0x00000200,seq=0x41): ESP(spi=0x00000201,seq=0x41), length 56    
18:46:11.232615 IP hope > charity.progclub.org: AH(spi=0x00000300,seq=0x23): ESP(spi=0x00000301,seq=0x23), length 96    
18:46:11.233099 IP charity.progclub.org > hope: AH(spi=0x00000200,seq=0x42): ESP(spi=0x00000201,seq=0x42), length 56    
18:46:11.233205 IP charity.progclub.org > hope: AH(spi=0x00000200,seq=0x43): ESP(spi=0x00000201,seq=0x43), length 96    
However, the SSH session just hangs there and "nothing happens". I have to press Ctrl+C to cancel out of the attempted SSH connection.
So, I've configured IPSec on each host, and according to tcpdump IPSec is being used, however the test connection just hangs. I'm at a loss for what to do next. How can I debug this problem? What have I missed in my configuration? Thanks in advance for any suggestions.
p.s. I'll change the keys from these ones I've posted publicly.
 John Elliot V
 John Elliot VUser contributions licensed under CC BY-SA 3.0