OpenSwan and Cisco PIX: ISAKMP:not zero on reserved payload 5

Now this was another long night: I was trying to create IPSec tunnel between Cisco PIX and Ubuntu-based router with OpenSwan. At first, I was unable to estabilish ISAKMP communication as PIX always rejected it with SA not acceptable. That was solved with adding


into ipsec.conf configuration file. And once I got though this I came to thegreat showstopper, that is mentioned in the title. Cisco and various forums state, that my pre-shared key was wrong (but it wasn’t), that I should do clear crypto sa on the PIX (but it didn’t help), or that my access list was wrong (but they seemed right). After giving up and getting some sleep instead I decided to change the encryption (I used AES with SHA1). I added new policy for 3DES with MD5 and suddenly the message was gone!

But there was a new one, saying Proxy Identities Not Supported, and finally I found where I should have had instead of – in the ipsec.conf.

And from that time on, the tunnel works perfectly.

One more note:

After you’re happy that your tunnel has been estabilished and you’re re looking forward to trying it out, you may get disappointed that no pings to remote internal network will work, and that ssh will return No route to host.The secret ingredient is to use internal network interface as source, so use

ping remoteinternalhost -I routerinternalinterfaceip
ssh -b routerinternalinterfaceip remoteinternalhost
July 3rd, 2008 23:01

add leftsourceip=routerinternalinterfaceip to your setup and you will not need to spewcify any internal ip’s anymore