Oodles of 529 Logon Failures every 2:00 AM

[-]

Hello,

My Windows 2000 domain is getting an error every night at 2AM because it
can't lock out the Administrator account. Yes, exactly; "why is it being
told to lock out in the first place?" I don't think we're under attack
because it is every night at the same time and because I have found some
information which may shed some light on it.

It seems that at 2:00 AM some process happens that all of the local
administrator accounts on the servers get a failed login to their local
machine. The domain registers these logon failures I suppose because the
machine itself is a member of the domain. The really weird thing is that
the "logon type" shows as type 3, network. How can a local account have a
network logon to its own machine?

More wierdness, wherever the local admin account of the server has been
changed, _that_ name shows up with the failed 529. The domain name is
_always_ the name of the local server, the AD domain is not referenced even
once in all 200 of the 529's.

Something... is causing these failed local admin logins to happen every
night at 2AM on servers. I think that's why the domain admin account is
receiving a call to get locked out is; because the domain is confusing the
local admin accounts with the domain admin account, and thinking that _it_
is the culprit.

The first thing we're going to do is rename the domain admin account (yes I
know I should have done this a long time ago, but there are services,
scheduled tasks, etc. running under that name that I have to track down and
remediate before I change it).

The next thing I will do is I will check with our server team about nightly
processes/tasks that may be occurring at 2AM, but I wonder if there is
something in the undulations of AD itself that is triggering this, such as a
master browser election.

If anyone can shed any light or has experienced something similar, I am open
to any advice you could give.

Thanks a bunch!!

 

[Roger Abell]

Wow, that is a massively confused situation (and large but
mostly reasonable x-post I leave untouched)

I am inlining some comments that may shed some light, and
hope (for your sake) that others add more.

--
Roger Abell
Microsoft MVP (Windows Server System: Security)
MCSE (W2k3,W2k,Nt4) MCDBA

Hello,

My Windows 2000 domain is getting an error every night at 2AM because it
can't lock out the Administrator account.

So you are saying that the domainname\administrator account
is being (or rather the attempt is made to have it) locked out.

Yes, exactly; "why is it being
told to lock out in the first place?"

because the invalid login count threshold is reached within
the time allowed, as you well know

I don't think we're under attack
because it is every night at the same time and because I have found some
information which may shed some light on it.

Gut level feeling are often right, but sometimes wrong.
Why do you not think it an attack?

It seems that at 2:00 AM some process happens that all of the local
administrator accounts on the servers get a failed login to their local
machine.

So, the process is attempting to log in with the domainname\administrator
and with each machine\administrator account

The domain registers these logon failures I suppose because the
machine itself is a member of the domain.

No. The login attempts are logged where authentication is processed.
For machine\administrator this is on machine, for domain\administrator
this is on a domain controller

The really weird thing is that
the "logon type" shows as type 3, network. ??

How can a local account have a
network logon to its own machine?

Nothing strange here. If login is by use of a network based
access. First, it sounds like at each machine, the process may
be attempting to use in turn machine\admininstrator and also
domain\administrator. This likely originates on some machine
other than the one targetted, but it could orginate there and still
be login type 3.

More wierdness, wherever the local admin account of the server has been
changed, _that_ name shows up with the failed 529.

This indicates that either you have not tightened the machines (and if the
domain\administrator account falls into this camp, tightened the domain)
so that it does not allow enumeration of accounts; or, that the process that
is behind the behavior has access to a valid login so that it can enumerate
account names non-anonymously. If you can query against the SAM of
account info, it is not hard to know which are admins.

The domain name is
_always_ the name of the local server, the AD domain is not referenced even
once in all 200 of the 529's.

I do not follow what that said, as it seems to say one thing and then
say that it is not what was just said.

Something... is causing these failed local admin logins to happen every
night at 2AM on servers.

Yes. As they say on Mission Impossible, your task, should you choose
to accept it, is . . .

I think that's why the domain admin account is
receiving a call to get locked out is; because the domain is confusing the
local admin accounts with the domain admin account, and thinking that _it_
is the culprit.

Again, I got lost on what that was saying. "The domain admin account
is receiving a call to get locked out is . . ." ??

The first thing we're going to do is rename the domain admin account (yes I
know I should have done this a long time ago, but there are services,
scheduled tasks, etc. running under that name that I have to track down and
remediate before I change it).

Not just the domain\administrator account, but each machine\administrator
account (and, ideally not all to the same thing).
Reset passwords while at it.

The next thing I will do is I will check with our server team about nightly
processes/tasks that may be occurring at 2AM,

Excellent idea, especially now that it is apparent that there are
evidently admins of servers in your environment doing things
of which you may have no awareness.
Also, you may want to consider reviewing successful logins
onto domain accounts, or onto the servers, at about the same
time, or in the interval before the event begins.
Do you have uplevel machines ? The event logs on uplevels
will provide info on the originating IP for the failed attempts.

but I wonder if there is
something in the undulations of AD itself that is triggering this,

no, not that I can thing of, but it certainly could be programmed
to do so, just not "as shipped"

such as a
master browser election.

That is pre-AD, and is non-authenticated.
My first thought is dumb backup software someone is trying out
and did not configure, or something like Nessus that someone has
decided would be good to turn loose at 2 am to scan about.

If anyone can shed any light or has experienced something similar, I am open
to any advice you could give.

Thanks a bunch!!

Good luck. Collect the dominoes and the picture will point
your nose in the right direction.

 

[Steven L Umbach]

I think your best bet would be to try and track down what is happeneing at 2:00AM.
See if the failed type 3 logons are originating from the same computer and then see
what is happening on that computer checking Scheduled Tasks and AT tasks by entering
AT on the command line and also enabling auditing of process tracking on it. It would
have nothing to do with the browser elections. I beleive if the domain name and the
server name are the same in the failed logon that means that the failed logon was
against a local computer account rather than domain account. For domain account
failed logons it may help to refer to the link below and use netlogon logging to find
the computer or computers causing these failed logons. It also has a lot of good info
on tracking down failed logons and common reasons why they happen. I would also
suggest that you try http://www.eventid.net for Event ID 529 to see if you find
anyhting helpful there. --- Steve

http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/security/bpactlck.mspx
http://eventid.net/display.asp?eventid=529&eventno=1&source=Security&phase=1 ---
Eventid.net for ID 529.