L2TP/IPSEC - error 678

  • Thread starter Thread starter JJ
  • Start date Start date
J

JJ

I am in the middle of a win2003 RAS rollout...with an end goal of L2TP/IPSEC
for both VPN and wireless connections (802.1x). I've waded through the PKI
setup...certs are issued to my IAS/RAS servers and my test client machine.

Servers are all win2003...client is winXP (sp2). RAS server has a public IP
(firewalled) and a private IP (for corporate LAN)...authentication is via
IAS installed on win2003 DC's...client is using standard dial-up (no NAT).

I can establish a VPN connection through PPTP...with either CHAP or
EAP-TLS...with no problems.

When I attempt to connect via L2TP/IPSEC I consistently get 678 errors
(server did not respond)...this is the case for both preshared key and
certificate attempts.

When I attempt the L2TP connection it behaves as if it were a firewall
problem...client sends out an L2TP request on 1701...and then seemingly
nothing happens...error 678 server did not respond. However...I have tested
with the client and RAS server on the same (public) subnet...as well as
opening all traffic to/from the RAS server from another known public IP. So
I am fairly confident it is not a firewall issue.

The fact that PPTP works with EAP-TLS would seem to imply that it is not a
certificate related problem. As would the fact that L2TP also fails with
preshared key attempts.

I've not been this stumped in quite some time...would appreciate advice on
where to focus troubleshooting efforts.
 
It is pretty hard to troubleshoot from here, but some questions
that might help you:

1) Do the clients actually have the CS trust certificate
for the issuing (to the RRAS server) server, and their
own CLIENT certificate (which is normally on the
Smartcard for EAP-TLS but could be in the client
store I suppose)?

2) Does the RRAS server have both it's own server
certificate (good for IPSec) and it's server trust cert
for the Certificate Server?
 
It is pretty hard to troubleshoot from here, but some questions
that might help you:

1) Do the clients actually have the CS trust certificate
for the issuing (to the RRAS server) server, and their
own CLIENT certificate (which is normally on the
Smartcard for EAP-TLS but could be in the client
store I suppose)?
Click to expand...

Client has a user certificate and a computer certificate...in local
store. Certs share a common root chain with those on servers.
2) Does the RRAS server have both it's own server
certificate (good for IPSec) and it's server trust cert
for the Certificate Server?
Click to expand...

RRAS and IAS servers have computer certificates in local store.

Are there other certificates I need to have in place? I saw the
IPSEC cert templates but wasn't sure if/which I needed and where
to place them.
 
JJ said:
Client has a user certificate and a computer certificate...in local
store. Certs share a common root chain with those on servers.


RRAS and IAS servers have computer certificates in local store.

Are there other certificates I need to have in place? I saw the
IPSEC cert templates but wasn't sure if/which I needed and where
to place them.
Click to expand...

L2TP is using IPSec for the actual encryption so
that has to be available as a cert type on each side
and with the trusts (for at least the opposite side.)

Chances are the certs are issued by the same CA
so in practice this probably means the same trust
cert.
 
L2TP is using IPSec for the actual encryption so
that has to be available as a cert type on each side
and with the trusts (for at least the opposite side.)

Chances are the certs are issued by the same CA
so in practice this probably means the same trust
cert.
Click to expand...

is this true regardles of whether i'm using EAP or
preshared keys to authenticate?

i'll have to give this a try...do you know which cert
i need to issue? i have a choice of IPSEC and
IPSEC (offline request).
 
L2TP is using IPSec for the actual encryption so
that has to be available as a cert type on each side
and with the trusts (for at least the opposite side.)

Chances are the certs are issued by the same CA
so in practice this probably means the same trust
cert.
Click to expand...

i added an IPSEC cert to my RAS server and XP client.
still no joy...error 678...server did not respond.
 
JJ said:
is this true regardles of whether i'm using EAP or
preshared keys to authenticate?
Click to expand...

No, preshared secret doesn't require certificates.

That is easy to setup with pure IPSec (LAN style)
but how did you set that with RRAS?
i'll have to give this a try...do you know which cert
i need to issue? i have a choice of IPSEC and
IPSEC (offline request).
Click to expand...

Those aren't different certs, just different ways to
issue them (I believe.)
 
Have you tried the following in the built-in help:

[ RRAS checklist ]

[ L2TP checklist ]

Similar with "checklist" -- there are checklists for
every significant subsystem.

Also this is worth searching:

[ checklists ]

for a CHECKLIST of checklists...
 
Herb Martin said:
No, preshared secret doesn't require certificates.
Click to expand...

In that case I'm barking up the wrong tree with certificate
troubleshooting...I can't even establish a connection with preshared keys.
This is true if I test via dialup or with a client on the same subnet as the
RRAS.
That is easy to setup with pure IPSec (LAN style)
but how did you set that with RRAS?
Click to expand...

In RRAS on win2003...RRAS properties...security tab...'allow custom ipsec
policy for l2tp connection'...type in a preshared key in the provided field.
Those aren't different certs, just different ways to
issue them (I believe.)
Click to expand...

thanks for the clarification.
 
Herb Martin said:
Have you tried the following in the built-in help:

[ RRAS checklist ]

[ L2TP checklist ]

Similar with "checklist" -- there are checklists for
every significant subsystem.

Also this is worth searching:

[ checklists ]

for a CHECKLIST of checklists...
Click to expand...

thanks for your efforts...yes...i've reviewed most of the built-in help
documentation to no avail.

i thought i was close to getting this working a few nights back. i was
getting a connection...at which point the server was issuing a deny (default
for my IAS connection policy was set to deny...changed to grant and i then
got a 'client does not support this encryption level' or a similar error).

of course in all the attempts i've made since then i am now not sure if the
'near' connection that night was made using EAP or CHAP over L2TP...or if i
had PPTP enabled at the server as well and it was simply connecting there
first...i am almost certain i disabled the PPTP ports on the RRAS server.
either way...that was the closest i've been to getting this working...its
been 'server did not respond' ever since.

so to review...in hopes that anyone else can shed some light...here is my
current setup:

IAS/DC/CA - computer certificate installed in local store...server resides
on corpLAN (private)

RRAS - computer certificate installed in local store...1 PPTP and 1 L2TP
port active for testing purposes...authentication via IAS...one nic on
corpLAN...one with a public IP address

XPsp2 - client test machine...user and computer certificate installed in
local store

PPTP connection works like a champ...with both CHAP and EAP authentication.
L2TP/IPSEC fails...with both preshared keys or EAP...error 678 'server did
not respond'.

i gotta belive there's something i've missed here...but i'll be damned if i
can determine what it is even after reading through all the how-to's and
checklists...and here i thought setting up the PKI would be the hard part of
this process. boy was i wrong.
 
In RRAS on win2003...RRAS properties...security tab...'allow custom ipsec
policy for l2tp connection'...type in a preshared key in the provided field.
Click to expand...

Did you also set the client side?
 
Herb Martin said:
Did you also set the client side?
Click to expand...

yes...connection type forced to l2tp/ipsec in networking tab...preshared key
set in ipsec settings under security tab.
 
JJ said:
yes...connection type forced to l2tp/ipsec in networking tab...preshared key
set in ipsec settings under security tab.
Click to expand...

I have never used pre-shared secret for Dial/VPN
but only for LAN type IPSec -- I wasn't even sure
you could get to the settings (although I am not
surprised now that you tell me it worked ok.)
 
I am in the middle of a win2003 RAS rollout...with an end goal of L2TP/IPSEC
for both VPN and wireless connections (802.1x). I've waded through the PKI
setup...certs are issued to my IAS/RAS servers and my test client machine.

Servers are all win2003...client is winXP (sp2). RAS server has a public IP
(firewalled) and a private IP (for corporate LAN)...authentication is via
IAS installed on win2003 DC's...client is using standard dial-up (no NAT).

I can establish a VPN connection through PPTP...with either CHAP or
EAP-TLS...with no problems.

When I attempt to connect via L2TP/IPSEC I consistently get 678 errors
(server did not respond)...this is the case for both preshared key and
certificate attempts.

When I attempt the L2TP connection it behaves as if it were a firewall
problem...client sends out an L2TP request on 1701...and then seemingly
nothing happens...error 678 server did not respond. However...I have tested
with the client and RAS server on the same (public) subnet...as well as
opening all traffic to/from the RAS server from another known public IP. So
I am fairly confident it is not a firewall issue.

The fact that PPTP works with EAP-TLS would seem to imply that it is not a
certificate related problem. As would the fact that L2TP also fails with
preshared key attempts.

I've not been this stumped in quite some time...would appreciate advice on
where to focus troubleshooting efforts.
Click to expand...

problem (FINALLY) solved...and i hope one of the MS guys can/will
comment on this. IPSEC was disabled in the registry on my client
machine. not sure how/why this was so...perhaps a sp2 change on xp?

anyway...the key i needed to change to get L2TP/IPSEC working:

HKLM\System\CurrentControlSet\Services\RasMan\Parameters\ProhibitIpSec

changed from 1 (default?!?) to 0...problem solved.
 
problem (FINALLY) solved...and i hope one of the MS guys can/will
comment on this. IPSEC was disabled in the registry on my client
machine. not sure how/why this was so...perhaps a sp2 change on xp?
Click to expand...

Glad you solved it.
anyway...the key i needed to change to get L2TP/IPSEC working:

HKLM\System\CurrentControlSet\Services\RasMan\Parameters\ProhibitIpSec

changed from 1 (default?!?) to 0...problem solved.
Click to expand...

I really don't think that is the default -- how did you
find that?
 
Herb Martin said:
Glad you solved it.


I really don't think that is the default -- how did you
find that?
Click to expand...

I stumbled on out of desperation...I went to ADD the key only to find it was
already present. I checked some other machines and it is clearly NOT the
default...as a matter of fact the key isn't even present.

I strongly suspect a 3rd party VPN client (WatchGuard branded SafeNet) was
to blame...the registry setting in question exists on all of the machines
that have this particular VPN client installed...which includes the machine
I was using for testing.
 
Back
Top