MacOSX 10.5 as VPN server

I am running some MacOSX Servers, and got a question from a friend whether I could terminate VPN connections that he was interested in. He runs Windows XP, and according to what one can find on the web, this should be no problems at all. Now I know that was really last famous words, but at least it works now. Let me tell you about the not so well documented issues.

First of all, do activate the VPN service in Server Admin. As you can find with Google, Windows XP uses PPTP, so that is where the interesting stuff happens.

Enable PPTP, and allow 40-bit encryption keys in addition to 128-bit, use MS-CHAPv2 as the PPP Authentication mechanism.

First not very well documented issue is that the start and end IP addresses must indicate an interval of IP addresses the VPN terminating server see locally (that it can proxy arp for). I.e. IP addresses on the same subnet as the external interface on a host (like mine) that only have one IP address. Btw, I have always taken for granted this only works with IPv4. My host have IPv6 as well, but all of this is only IPv4.

So, check for an interval of IP addresses on the same subnet as your VPN server itself, and indicate that interval in the PPTP settings. Getting the VPN termination box to also do NAT via the NAT service is something I have not succeeded with. I was hoping the VPN would end up as a virtual interface in the NAT settings, but no, that is not happening.

Second thing that is not very well documented is that what happens part from starting the VPN service is that first time you launch VPN (or maybe when you activate the service), two (2) more things are happening:

  • An entry is added to your OD server

  • An entry is added to the System Keychain

I found I had to remove these, and add both with the terminal, i.e. manually. As many things (unfortunately) in the MacOSX Server automatic GUI settings, they take for granted the scenario one have is pretty simplistic.

Because of this, before you start the VPN service (or after you stop it, as you of course have already started it when reading this), remove all users in OD that have the name of VPN MPPE Key Access User. Sometimes you might have managed to get more than one of these, so although you are doing “Delete” in Workgroup Manager, it might look like if nothing happens. Just do this until there is nothing visible there.

You should also remove all keys in the System Keychain (that you can access via Keychain Access) that have the name ending with com.apple.ras. Just remove them. As far as I understand, they are used for exactly this and nothing else.

When both OD entry and keys are removed, we have to create them again, and that is done with the command vpnaddkeyagentuser. The interesting thing here (that is not very well documented) is how to run the command. The manual page say you should do:


vpnaddkeyagentuser directory\_node\_path

And the example given is:


To add the keyagent user to the Open LDAP master on the local machine:

vpnaddkeyagentuser /LDAPv3/127.0.0.1

The key here is that if the LDAP master is not on the local machine, then you MUST include the hostname of the LDAP server in the path. Using IP address does NOT work. So if your LDAP server is ldap.example.com, your command is:


sudo /usr/sbin/vpnaddkeyagentuser /LDAPv3/ldap.example.com

You will then be prompted for the password that enable sudo, and then username and password for a user that has access to add users to the LDAP database. What is normally called the diradmin in other documentations. You can check with Keychain Access and Workgroup Manager afterwards that the key and the user was added.

If you do not get explicitly a prompt for the username and password for a directory admin user, something went wrong. Start over, and ensure you give the command correctly. There is no error messages at all.

That should do the trick, as long as you manage to configure XP correctly, something I leave as an exercise for the reader.