Windows – Can’t connect to L2TP IPsec VPN from Windows 10 but it works with macOS High Sierra

ipsecl2tpvpnwindows

I am trying to connect from a Windows 10 client to an Ubiquiti EdgeRouter VPN, I went through the instructions on https://help.ubnt.com/hc/en-us/articles/204950294-EdgeRouter-L2TP-IPsec-VPN-Server but whenever I try to connect to the VPN I get the following:

Can't connect to
The L2TP connection attempt failed because the security layer could not negotiate compatible parameters with the remote computer.

I did some research and changed the VPN security properties to allow Microsoft CHAP Version 2 (MS-CHAP v2) and also Challenge Handshake Authentication Protocol (CHAP) but it didn't seem to make a difference at all.

At the same time I have no problems whatsoever to connect from a macOS High Sierra client. So my guess is that there is something wrong on the client side (Windows VPN settings)? Can anybody help me out with this?

Best Answer

That error message probably means that there is some level of connection between Windows and the Ubiquiti but they failed to find a common encryption method. For phase1 (key exchange) Windows (version 1803) is proposing the following encryption methods (in this priority order):

  1. SHA1 + AES-CBC-256 + ECP384
  2. SHA1 + AES-CBC-128 + ECP256
  3. SHA1 + AES-CBC-256 + MODP2048
  4. SHA1 + 3DES-CBC + MODP2048
  5. SHA1 + 3DES-CBC + MODP1024

For phase2:

  1. SHA1 + AES-CBC-256
  2. SHA1 + AES-CBC-128

Make sure the Ubiquiti can accept one of these proposals so try to set the phase1 and phase2 proposals on the Ubiquiti to something like this.

set vpn ipsec ike-group FOO0 lifetime 86400
set vpn ipsec ike-group FOO0 proposal 1 dh-group 14
set vpn ipsec ike-group FOO0 proposal 1 encryption aes256
set vpn ipsec ike-group FOO0 proposal 1 hash sha1
set vpn ipsec esp-group FOO0 lifetime 43200
set vpn ipsec esp-group FOO0 pfs disable
set vpn ipsec esp-group FOO0 proposal 1 encryption aes128
set vpn ipsec esp-group FOO0 proposal 1 hash sha1

I would also enable NAT-Traversal too:

set vpn ipsec nat-traversal enable

I do not really know Ubiquiti but maybe this helps.