100% cpu usage for LSASS.EXE on DC intermittently, consistent inte

[Guest]

Apparently Last week (while I was gone of course) this issue started. I
noticed it on my return last friday. What I noticed is...

On my AD User Domain a DC which has PDC & GC roles spiked CPU at consistent
intervals. Looking at Task Manager, it was LSASS.EXE which was consuming CPU
during these spikes. No other DC (3 others in Root domain, 3 others in User
domain) exhibited this behavior. This DC has Dual hyperthreaded 2.4G
processors so I only noticed two of the four CPU threads spike during the
event. I did not time the event or interval on this machine.

To make sure it wasn't the machine mis-behaving I rebooted that DC.

On reboot that DC again functioned consistent with previous CPU behavior
patterns (no unusual spikes) but the issue moved to another DC in the User
domain.

On this DC (single non-hyperthreaded 1.4G cpu) the event appears to last 10
secs (that is LSASS.EXE comsumes 99% of cpu) and re-occurs about every 60
secs after the end of the last event. This DC has the domain infrastrure
role and is also a WINS and DNS server.

I thought I had seen something like this before when someone had attempted
to configure a SAMBA client, but had done so incorrectly. Note: I have known
SAMBA users correctly configured who appearently don't cause this problem
(they've been running for months/years against these same DCs). I just know
I've seen a similar event to this in the past, had a person tell me that had
a samba config problem, I've ask them to discontinue use and the problem went
away.

Only the CPU spikes, I don't see similar spikes in network, memory or drive
performance (performance monitor).

I've look on the Internet and see references to WORMS doing this, but
appearently only one machine must be infected or I'd think more than one DC
would see this issue and the events would not be so consistent with more than
one source.

I've put a sniffer on the appropriate switch and I've looked at the capture,
but since this is a functioning DC and I'm not sure what I'm looking for, I
can't seem to separate out the 'wheat from the chaff'.

Help, I need someone to give me a clue on what I need to do to track this
issue down.

thanks much. - bill.

 

[Phillip Windell]

I've look on the Internet and see references to WORMS doing this, but
appearently only one machine must be infected or I'd think more than one DC
would see this issue and the events would not be so consistent with more than
one source.

That is the wrong thing to expect. It is probably infected. Everytime I
have heard of this happening, without exception,...it was infected.

 

[Guest]

Phillip Windell [MCP, MVP, CCNA]
That is the wrong thing to expect. It is probably infected. Everytime I
have heard of this happening, without exception,...it was infected.

Thanks for your response Phillip, but...

I doubt any of the DC's are infected themselves. They are not logged into
interactively accept to do DC work (no email, no web). They always have the
latest security patches applied. If it is a WORM on a client machine, very
possible, (like MS-Blaster, etc) it must be a worm specific to hitting a
single DC. Again note, I don't see this behavior on any other machine (other
DC's, member server, or clients) which I would expect to see in the case of a
worm randomly walking the internal address spaces.

Anyone have any more insight on what to look for here.
Anyone tell me what to look for in my sniffer captures.
tks. - bill.

 

[Guest]

There is an LSASS worm I think. I think I also saw this with McAfee AV.

Phillip Windell [MCP, MVP, CCNA]
That is the wrong thing to expect. It is probably infected. Everytime I
have heard of this happening, without exception,...it was infected.

Thanks for your response Phillip, but...

I doubt any of the DC's are infected themselves. They are not logged into
interactively accept to do DC work (no email, no web). They always have the
latest security patches applied. If it is a WORM on a client machine, very
possible, (like MS-Blaster, etc) it must be a worm specific to hitting a
single DC. Again note, I don't see this behavior on any other machine (other
DC's, member server, or clients) which I would expect to see in the case of a
worm randomly walking the internal address spaces.

Anyone have any more insight on what to look for here.
Anyone tell me what to look for in my sniffer captures.
tks. - bill.

 

[Guest]

I've look on the Internet and see references to WORMS doing this, but

Phillip Windell [MCP, MVP, CCNA]
That is the wrong thing to expect. It is probably infected. Everytime I
have heard of this happening, without exception,...it was infected.

Thanks for your response Phillip, but...

I doubt any of the DC's are infected themselves. They are not logged into
interactively accept to do DC work (no email, no web). They always have the
latest security patches applied. If it is a WORM on a client machine, very
possible, (like MS-Blaster, etc) it must be a worm specific to hitting a
single DC. Again note, I don't see this behavior on any other machine (other
DC's, member server, or clients) which I would expect to see in the case of a
worm randomly walking the internal address spaces.

Anyone have any more insight on what to look for here.
Anyone tell me what to look for in my sniffer captures.

:
There is an LSASS worm I think. I think I also saw this with McAfee AV.

Ok, let assume it's a worm. (I kind of wonder if it is, because a worm
would likely attack more than one DC at a time or at least move from DC to
DC, that's the behavior we've seen with netbios worms in the past).

So, now I'm back to my second question above.

What do I need to look for in my sniffer captures (or using other tools) to
find the offending machine. We have 2000+ machines in our lAN and we are
closely tied (i.e. have Netbios ports open) to thousands more in our sister
schools.

Anyone know how to find the offending machine(s)?

 

[Michael D. Ober]

I have also seen this when the physical NIC fails.

Mike Ober.

There is an LSASS worm I think. I think I also saw this with McAfee AV.

I've look on the Internet and see references to WORMS doing this,

but

Phillip Windell [MCP, MVP, CCNA]
That is the wrong thing to expect. It is probably infected. Everytime I
have heard of this happening, without exception,...it was infected.

Thanks for your response Phillip, but...

I doubt any of the DC's are infected themselves. They are not logged into
interactively accept to do DC work (no email, no web). They always have the
latest security patches applied. If it is a WORM on a client machine, very
possible, (like MS-Blaster, etc) it must be a worm specific to hitting a
single DC. Again note, I don't see this behavior on any other machine (other
DC's, member server, or clients) which I would expect to see in the case of a
worm randomly walking the internal address spaces.

Anyone have any more insight on what to look for here.
Anyone tell me what to look for in my sniffer captures.
tks. - bill.

 

[Phillip Windell]

What do I need to look for in my sniffer captures (or using other tools) to
find the offending machine. We have 2000+ machines in our lAN and we are
closely tied (i.e. have Netbios ports open) to thousands more in our sister
schools.

You don't. You must scan for them with decent AV Software. If you are not
runnig with that kind of protection in a school environment already to begin
with you are being suicidal. School networks are the worst networks on
Earth for being a virus haven, a spyware haven, and about everything else
that is bad on a network.

 

[Guest]

I have also seen this when the physical NIC fails.

Michael, thanks for the input. however if you read the original post, you
will see that this issue moved from one DC to another after a reboot, this is
not a hardware issue.

- bill.

 

[Guest]

You don't. You must scan for them with decent AV Software. If you are not
runnig with that kind of protection in a school environment already to begin
with you are being suicidal. School networks are the worst networks on
Earth for being a virus haven, a spyware haven, and about everything else
that is bad on a network.

Phillip, an important comment, but you missed the point. We ARE running AV
software on our clients. However, it only takes one client (mis-configured,
un-configured, never configured) to be a source for a WORM as you well know.

You may be hinting that I need to install AV software on my DC's, however, I
don't know that Microsoft supports DC's with AV installed. Also, I'm not
sure AV software would detect a WORM intrusion attempt. That seems more like
the work of Firewall or IDS software.

Again, I don't believe the DC's themselves are contaminated, one of them is
just being attacked (or at least unusually accessed). I'd love to believe
that if the source was one of the campus (or remote campus) PCs that it would
have AV installed and that the user was also competent to note the infection.
But since I don't maintain those thousands of desktops, their is no way I
can confirm they are all clean or the users are all competent.

What I do know is that something is sending something to one of my DCs. I'm
hoping that an MS representative reads this thread and leads me in the right
direction.

I'm sure DCs are susceptable to this in any large organization and the
Microsoft is aware of this. Otherwise they wouldn't have added the Firewall
option to SP1 in W2k3. - bill

 

[Phillip Windell]

You may be hinting that I need to install AV software on my DC's, however, I
don't know that Microsoft supports DC's with AV installed. Also, I'm not
sure AV software would detect a WORM intrusion attempt.

That is exactly what I am saying. Put it on the DCs. The only Server MS
makes that needs special consideration is Exchange (maybe SQL Server).
Worms are specifically detected by AV software,..worms are a variation of a
virus not an "intrusion".

That seems more like the work of Firewall or IDS software.

No. Completely the opposite.
Those primarily protect from "hacking" not Virus/Worms.
Some have added the ability as an "extra" feature, but it is not thier
primary purpose.

Again, I don't believe the DC's themselves are contaminated,

I have a hard time believing that it is not contaminated at this point. You
won't know till you put the AV on it and do the scan.

But since I don't maintain those thousands of desktops, their is no way I
can confirm they are all clean or the users are all competent.

All the more reason to put AV on the Server.

What I do know is that something is sending something to one of my DCs. I'm
hoping that an MS representative reads this thread and leads me in the right
direction.

It think it is originating on the DC by this point.

I'm sure DCs are susceptable to this in any large organization and the
Microsoft is aware of this. Otherwise they wouldn't have added the Firewall
option to SP1 in W2k3.

That is not what firewalls are for. Firewalls are for Hackers. AV Software
is for virus/worms.

At this point all I am going to be doing is repeating myself over and over.
If you aren't going to listen, there isn't anything else I can do. If you
do the AV Scan and nothing comes up then we can deal with that, but until
that is proven I am dead-set that you need to do what I suggested.

 

[Guest]

That is exactly what I am saying. Put it on the DCs. The only Server MS

I have a hard time believing that it is not contaminated at this point. You
won't know till you put the AV on it and do the scan.

It think it is originating on the DC by this point.

At this point all I am going to be doing is repeating myself over and over.
If you aren't going to listen, there isn't anything else I can do. If you
do the AV Scan and nothing comes up then we can deal with that, but until
that is proven I am dead-set that you need to do what I suggested.

Phillip, don't be frustrated. I'm already frustrated and one frusrated
person in this thread is enough.

I've done a simple procedure that appears to have eliminated the issue. But
I feel confident that anything has changed for awhile.

Please bare with me for 24hours and I'll get back to you on what has
happened between now and then with this issue.

I do appreciate your help, and I do believe you are doing the best you can
with the info you have, but I have more background, more experience with this
site, I better understand the issues going on here.

I query the news groups only to gain additional info that I don't have/don't
know.

I don't expect you to be up to speed with the intimate details of my site or
the conditions there in, with just a few paragraphs. tks. - bill.

 

[Phillip Windell]

Phillip, don't be frustrated. I'm already frustrated and one frusrated
person in this thread is enough.

I'm fine If we were talking in person I wouldn't have sounded
frustrated. Email message don't communicate emotions very well.

I've done a simple procedure that appears to have eliminated the issue. But
I feel confident that anything has changed for awhile.

Please bare with me for 24hours and I'll get back to you on what has
happened between now and then with this issue.

Sounds good. We'll see what happens.

 

[Guest]

Sounds good. We'll see what happens.

Last night I 'dis-connected' the specific DC from the network for 5 mins. I
didn't reboot (i.e. didn't clear memory of any executing programs). For the
next 15 hours the server's behavior was normal (usual per previous
experience).

When I checked on the 16th hour LSASS.EXE was again running at 98% for 10
sec, then there was an interval of 60 secs when cpu was normal (pretty much
idle on this DC), then the cycle repeats, just like the event looked before.

Tonight after hours I intend to reboot this DC to see if it forces this
'event' to move to another DC just like the previous reboot did last weekend
when I caused the event to move to this DC...

I still don't see any evidence of worm activity in the sniffer capture logs
(in either direction). I also don't see a lot of difference in the sniffer
logs between when the event was not occuring and when it is occuring (that is
there is no spike in either network traffic or communication patterns (hosts
contacted) when CPU spikes as I would expect with a network worm).

As a standard practice I don't load any 3rd party software on my DCs. But
I'm considering putting up a temporary DC (think of it as a honey-pot) with
an AV as you suggest. If I do can you answer the following.

1) If I then log a pay call to MS on this after I install an AV on the DC,
will microsoft still support it.

2) We have a site license for McAfee AV (currently at version 8.0i) is that
a supported AV on a DC.

3) Are there any known requirements (features to turn on/off) to installing
an AV on a DC to be considered still running a 'supported' installation.

I intend on moving my W2K AD domain to W2K3 this summer and if installing an
AV on DCs is a recommended option, I'd like to know that before I start
building new servers. I prefer new builds to upgrades.

Finally, even though you don't want to entertain this option. What TCP/UDP
port does in-bound LSASS.EXE communicate (listen) on - I'd like to filter my
sniffer captures by this port to see who is kicking off that process by
remotely sending packets to this server.

 

[Phillip Windell]

As a standard practice I don't load any 3rd party software on my DCs. But
I'm considering putting up a temporary DC (think of it as a honey-pot) with
an AV as you suggest. If I do can you answer the following.

1) If I then log a pay call to MS on this after I install an AV on the DC,
will microsoft still support it.

Yes. They would think you were nuts for not running it on the DC. The DC
should be the first machine on the network with AV.

2) We have a site license for McAfee AV (currently at version 8.0i) is that
a supported AV on a DC.

I don't use McAfee and don't know much about them.

3) Are there any known requirements (features to turn on/off) to installing
an AV on a DC to be considered still running a 'supported' installation.

None that I am aware of. I have been running Symantec AV Corp Edition on
every single machine in our system for over four years. Started with version
7,...am now up to version 9.

Finally, even though you don't want to entertain this option. What TCP/UDP
port does in-bound LSASS.EXE communicate (listen) on - I'd like to filter

my

I have no idea.

 

[Guest]

TCP/UDP port does in-bound LSASS.EXE communicate (listen) on - I'd like to filter
my I have no idea.

Phillip, I'm going to submit this question to the security group. I'm also
putting up a IDS in the subnet to see what it sees. Thanks for your help. -
bill.

 

[Phillip Windell]

Sounds good! Good luck with it Bill!

 

[Guest]

Sorry I haven't been able to monitor this thread. I understand not running
an AV product on certain server types but it is certainly recommended and
supported. At the very least it should be installed and dissabled except for
manual scans. - More below:

thanks for the additional input. - bill