ASPNET_WP.EXE Hangs (W2K/IIS5/ASP.NET 1.1) With Minimal Load

  • Thread starter Thread starter Ken Barrett
  • Start date Start date
K

Ken Barrett

I apologize in advance for cross-posting, but I have noted that others
have experienced similar issues and was hoping that someone could
help.

Briefly, I am working on an ASP.NET content management application
written in C#. running on .NET Framework 1.1/W2K/IIS5. The
application uses the Microsoft WebService Behavior to generate a tree
of "folders" in a frame on the left hand side. The user can click on
a "folder", and the contents are rendered in the top frame on the
right. The user can then click on individual items (e.g. documents)
and they are displayed in the bottom frame on the right. The objects
themselves are stored in a strongly typed session object, and I am
starting to suspect that might be the issue.

Under minimal load (i.e. 1 user), the application runs for a short
while, and then the aspnet_wp.exe process hangs. It is never at the
same point (i.e. different users experience the hang at different
points), but the one thing that is fairly common is that task manager
shows that the CPU is happy (i.e. running around 3%-12%), and the
aspnet_wp.exe process has consumed approximately 65MB of memory. The
user can click on links in the browser (IE 6.0 SP1), and our
javascript runs, but no requests are logged on the server (I have used
Perfmon to verify that no requests are queued).

The IIS log shows a lot of 304 status codes for images, with a few 500
10054 status codes mixed in. In addition, there are sporadic
instances of a status code of 500 with no other information (i.e. just
the time and the value 500). However, at the point that the
application hangs, the log shows a status of 200 for the last request
handled.

I have used IISState, AD+ and WinDbg to try an pinpoint what is going
on. I have followed the articles at Microsoft on using WinDbg and SOS
to pinpoint the cause of the hang. The !threads command shows that we
have 1 dead thread, and further analysis shows that all of the threads
have a Suspend value of 1. Which, according to the articles, means
they will not run. However, I cannot for the life of me figure out
why or how. I could use some help deciphering the dumps and logs,
since I am new to WinDbg. I didn't include the log here, but I can in
future posts.

Any help that any can offer is greatly appreciated.

Thanks in advance,
Ken Barrett
MCP .NET
 
Pat said:
Can you post the IISState log so that we can see?

Pat

I posted it at Developersdex.com, but I will re-post below.

As I mentioned in the other post, I have an ADPlus_ASPNET log that is
152K, if it helps.

Thanks again,
Ken Barrett
MCP .NET


IISState Log
------------
Opened log file 'C:\iisstate\output\IISState-2316.log'

***********************
Starting new log output
IISState version 3.2

Tue Nov 11 15:32:06 2003

OS = Windows 2000
Executable: aspnet_wp.exe
PID = 2316

Note: Thread times are formatted as HH:MM:SS.ms

***********************




Thread ID: 0
System Thread ID: 648
Kernel Time: 0:0:0.30
User Time: 0:0:0.100
Thread Type: Other
# ChildEBP RetAddr
00 0012f6d8 7c59a0ed ntdll!NtDelayExecution+0xb
01 0012f6f8 7c59a0b8 KERNEL32!SleepEx+0x32
02 0012f704 00442df0 KERNEL32!Sleep+0xb
03 0012ff60 0044408d aspnet_wp!wmain+0x3a1
04 0012ffc0 7c5987e7 aspnet_wp!wmainCRTStartup+0x143
05 0012fff0 00000000 KERNEL32!BaseProcessStart+0x3d




Thread ID: 1
System Thread ID: 684
Kernel Time: 0:0:2.82
User Time: 0:0:13.239
Thread Status: Thread is in a WAIT state.
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
succeeded
Loaded Son of Strike data table version 5 from
"C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll"
Thread 1
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 00b6ff54 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 00b6ff7c 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 00b6ff8c 792e9c0d KERNEL32!WaitForSingleObject+0xf
03 00b6ffb4 7c57b382
mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x127
04 00b6ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 2
System Thread ID: 804
Kernel Time: 0:0:0.10
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 2
Not a managed thread.


Begin System Thread Information

# ChildEBP RetAddr
00 00c6ff2c 7c59a0ed ntdll!NtDelayExecution+0xb
01 00c6ff4c 7c59a0b8 KERNEL32!SleepEx+0x32
02 00c6ff58 792e9ebe KERNEL32!Sleep+0xb
03 00c6ffb4 7c57b382 mscorwks!ThreadpoolMgr::GateThreadStart+0x54
04 00c6ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 3
System Thread ID: 238
Kernel Time: 0:0:0.20
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 00d6ff3c 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 00d6ff64 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 00d6ff74 00443020 KERNEL32!WaitForSingleObject+0xf
03 00d6ff80 7c34940f aspnet_wp!DoPingThread+0x10
04 00d6ffb4 7c57b382 MSVCR71!_endthread+0xaa
05 00d6ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 4
System Thread ID: 448
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 4
Not a managed thread.


Begin System Thread Information

# ChildEBP RetAddr
00 04f0fecc 7c59a059 ntdll!ZwWaitForMultipleObjects+0xb
01 04f0ff1c 7c599f6c KERNEL32!WaitForMultipleObjectsEx+0xea
02 04f0ff34 791d25d5 KERNEL32!WaitForMultipleObjects+0x17
03 04f0ff9c 791d254c mscorwks!DebuggerRCThread::MainLoop+0x90
04 04f0ffac 791d4d50 mscorwks!DebuggerRCThread::ThreadProc+0x68
05 04f0ffb4 7c57b382 mscorwks!DebuggerRCThread::ThreadProcStatic+0xb
06 04f0ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 5
System Thread ID: 534
Kernel Time: 0:0:0.30
User Time: 0:0:0.20
Thread Status: Thread is in a WAIT state.
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 5
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 070aff28 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 070aff50 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 070aff60 791bbe7a KERNEL32!WaitForSingleObject+0xf
03 070aff70 791bbea4 mscorwks!WaitForFinalizerEvent+0xde
04 070affb4 7c57b382 mscorwks!GCHeap::FinalizerThreadStart+0x96
05 070affec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 6
System Thread ID: 880
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 6
Not a managed thread.


Begin System Thread Information

# ChildEBP RetAddr
00 0763ff80 7c59a0ed ntdll!NtDelayExecution+0xb
01 0763ffa0 792ea483 KERNEL32!SleepEx+0x32
02 0763ffb4 7c57b382 mscorwks!ThreadpoolMgr::TimerThreadStart+0x30
03 0763ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 7
System Thread ID: 870
Kernel Time: 0:0:0.150
User Time: 0:0:2.613
Thread Status: Thread is in a WAIT state.
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 7
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 077cff58 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 077cff80 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 077cff90 792e91e0 KERNEL32!WaitForSingleObject+0xf
03 077cffb4 7c57b382 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x3a
04 077cffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 8
System Thread ID: 7c
Kernel Time: 0:0:0.20
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 8
Not a managed thread.


Begin System Thread Information

# ChildEBP RetAddr
00 08deff30 7c59a059 ntdll!ZwWaitForMultipleObjects+0xb
01 08deff80 792ea338 KERNEL32!WaitForMultipleObjectsEx+0xea
02 08deffb4 7c57b382 mscorwks!ThreadpoolMgr::WaitThreadStart+0x45
03 08deffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 9
System Thread ID: 798
Kernel Time: 0:0:0.120
User Time: 0:0:0.991
Thread Status: Thread is in a WAIT state.
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 9
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 099fff58 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 099fff80 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 099fff90 792e91e0 KERNEL32!WaitForSingleObject+0xf
03 099fffb4 7c57b382 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x3a
04 099fffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 10
System Thread ID: 548
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 09d3fe24 77d574d4 ntdll!NtReplyWaitReceivePortEx+0xb
01 09d3ff74 77d56d9e RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x74
02 09d3ff78 77d39a00 RPCRT4!RecvLotsaCallsWrapper+0x9
03 09d3ffa8 77d41c6c RPCRT4!BaseCachedThreadRoutine+0x4f
04 09d3ffb4 7c57b382 RPCRT4!ThreadStartRoutine+0x18
05 09d3ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 11
System Thread ID: 23c
Kernel Time: 0:0:0.20
User Time: 0:0:0.610
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 11
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 0a1eff54 7c585323 ntdll!NtRemoveIoCompletion+0xb
01 0a1eff80 792e9b2f KERNEL32!GetQueuedCompletionStatus+0x27
02 0a1effb4 7c57b382
mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x49
03 0a1effec 00000000 KERNEL32!BaseThreadStart+0x52

*****

Dump name is formatted as: PID-Timestamp.dmp

Creating C:\iisstate\output\2316-1068582728.dmp - mini user dump

*****

Closing open log file C:\iisstate\output\IISState-2316.log
 
This log shows an idle process (i.e. not hung or otherwise doing any work).
If the app was hung at the time, then it would appear to be a client side
issue.

Pat

Ken Barrett said:
"Pat [MSFT]" <[email protected]> wrote in message
Can you post the IISState log so that we can see?

Pat

I posted it at Developersdex.com, but I will re-post below.

As I mentioned in the other post, I have an ADPlus_ASPNET log that is
152K, if it helps.

Thanks again,
Ken Barrett
MCP .NET


IISState Log
------------
Opened log file 'C:\iisstate\output\IISState-2316.log'

***********************
Starting new log output
IISState version 3.2

Tue Nov 11 15:32:06 2003

OS = Windows 2000
Executable: aspnet_wp.exe
PID = 2316

Note: Thread times are formatted as HH:MM:SS.ms

***********************




Thread ID: 0
System Thread ID: 648
Kernel Time: 0:0:0.30
User Time: 0:0:0.100
Thread Type: Other
# ChildEBP RetAddr
00 0012f6d8 7c59a0ed ntdll!NtDelayExecution+0xb
01 0012f6f8 7c59a0b8 KERNEL32!SleepEx+0x32
02 0012f704 00442df0 KERNEL32!Sleep+0xb
03 0012ff60 0044408d aspnet_wp!wmain+0x3a1
04 0012ffc0 7c5987e7 aspnet_wp!wmainCRTStartup+0x143
05 0012fff0 00000000 KERNEL32!BaseProcessStart+0x3d




Thread ID: 1
System Thread ID: 684
Kernel Time: 0:0:2.82
User Time: 0:0:13.239
Thread Status: Thread is in a WAIT state.
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
succeeded
Loaded Son of Strike data table version 5 from
"C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll"
Thread 1
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 00b6ff54 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 00b6ff7c 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 00b6ff8c 792e9c0d KERNEL32!WaitForSingleObject+0xf
03 00b6ffb4 7c57b382
mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x127
04 00b6ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 2
System Thread ID: 804
Kernel Time: 0:0:0.10
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 2
Not a managed thread.


Begin System Thread Information

# ChildEBP RetAddr
00 00c6ff2c 7c59a0ed ntdll!NtDelayExecution+0xb
01 00c6ff4c 7c59a0b8 KERNEL32!SleepEx+0x32
02 00c6ff58 792e9ebe KERNEL32!Sleep+0xb
03 00c6ffb4 7c57b382 mscorwks!ThreadpoolMgr::GateThreadStart+0x54
04 00c6ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 3
System Thread ID: 238
Kernel Time: 0:0:0.20
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 00d6ff3c 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 00d6ff64 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 00d6ff74 00443020 KERNEL32!WaitForSingleObject+0xf
03 00d6ff80 7c34940f aspnet_wp!DoPingThread+0x10
04 00d6ffb4 7c57b382 MSVCR71!_endthread+0xaa
05 00d6ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 4
System Thread ID: 448
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 4
Not a managed thread.


Begin System Thread Information

# ChildEBP RetAddr
00 04f0fecc 7c59a059 ntdll!ZwWaitForMultipleObjects+0xb
01 04f0ff1c 7c599f6c KERNEL32!WaitForMultipleObjectsEx+0xea
02 04f0ff34 791d25d5 KERNEL32!WaitForMultipleObjects+0x17
03 04f0ff9c 791d254c mscorwks!DebuggerRCThread::MainLoop+0x90
04 04f0ffac 791d4d50 mscorwks!DebuggerRCThread::ThreadProc+0x68
05 04f0ffb4 7c57b382 mscorwks!DebuggerRCThread::ThreadProcStatic+0xb
06 04f0ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 5
System Thread ID: 534
Kernel Time: 0:0:0.30
User Time: 0:0:0.20
Thread Status: Thread is in a WAIT state.
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 5
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 070aff28 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 070aff50 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 070aff60 791bbe7a KERNEL32!WaitForSingleObject+0xf
03 070aff70 791bbea4 mscorwks!WaitForFinalizerEvent+0xde
04 070affb4 7c57b382 mscorwks!GCHeap::FinalizerThreadStart+0x96
05 070affec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 6
System Thread ID: 880
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 6
Not a managed thread.


Begin System Thread Information

# ChildEBP RetAddr
00 0763ff80 7c59a0ed ntdll!NtDelayExecution+0xb
01 0763ffa0 792ea483 KERNEL32!SleepEx+0x32
02 0763ffb4 7c57b382 mscorwks!ThreadpoolMgr::TimerThreadStart+0x30
03 0763ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 7
System Thread ID: 870
Kernel Time: 0:0:0.150
User Time: 0:0:2.613
Thread Status: Thread is in a WAIT state.
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 7
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 077cff58 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 077cff80 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 077cff90 792e91e0 KERNEL32!WaitForSingleObject+0xf
03 077cffb4 7c57b382 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x3a
04 077cffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 8
System Thread ID: 7c
Kernel Time: 0:0:0.20
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 8
Not a managed thread.


Begin System Thread Information

# ChildEBP RetAddr
00 08deff30 7c59a059 ntdll!ZwWaitForMultipleObjects+0xb
01 08deff80 792ea338 KERNEL32!WaitForMultipleObjectsEx+0xea
02 08deffb4 7c57b382 mscorwks!ThreadpoolMgr::WaitThreadStart+0x45
03 08deffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 9
System Thread ID: 798
Kernel Time: 0:0:0.120
User Time: 0:0:0.991
Thread Status: Thread is in a WAIT state.
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 9
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 099fff58 7c599e8e ntdll!NtWaitForSingleObject+0xb
01 099fff80 7c57b3d5 KERNEL32!WaitForSingleObjectEx+0x71
02 099fff90 792e91e0 KERNEL32!WaitForSingleObject+0xf
03 099fffb4 7c57b382 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x3a
04 099fffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 10
System Thread ID: 548
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 09d3fe24 77d574d4 ntdll!NtReplyWaitReceivePortEx+0xb
01 09d3ff74 77d56d9e RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x74
02 09d3ff78 77d39a00 RPCRT4!RecvLotsaCallsWrapper+0x9
03 09d3ffa8 77d41c6c RPCRT4!BaseCachedThreadRoutine+0x4f
04 09d3ffb4 7c57b382 RPCRT4!ThreadStartRoutine+0x18
05 09d3ffec 00000000 KERNEL32!BaseThreadStart+0x52




Thread ID: 11
System Thread ID: 23c
Kernel Time: 0:0:0.20
User Time: 0:0:0.610
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
Thread 11
ESP EIP


Begin System Thread Information

# ChildEBP RetAddr
00 0a1eff54 7c585323 ntdll!NtRemoveIoCompletion+0xb
01 0a1eff80 792e9b2f KERNEL32!GetQueuedCompletionStatus+0x27
02 0a1effb4 7c57b382
mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x49
03 0a1effec 00000000 KERNEL32!BaseThreadStart+0x52

*****

Dump name is formatted as: PID-Timestamp.dmp

Creating C:\iisstate\output\2316-1068582728.dmp - mini user dump

*****

Closing open log file C:\iisstate\output\IISState-2316.log
 
Pat said:
This log shows an idle process (i.e. not hung or otherwise doing any work).
If the app was hung at the time, then it would appear to be a client side
issue.

Pat

Pat,

Thanks for taking a look at the log. I find that odd because the
ASP.NET process was definitely hung at that point, and would not
respond to requests until I recycled it.

I wondered about the possibility of a client side issue, which is why
I applied the hotfix I mentioned (and, as I mentioned, it didn't fix
the issue).

One of my initial suspicions was in the approach used here to store
objects and DataTables in a strongly typed session object (not *my*
approach, just the one used here). However, I didn't pursue it too
far because the memory consumption just didn't seem that high to me.
Upon further investigation, this began to look like it may be the
culprit. I refactored and recoded a lot of what was being done with
this strongly typed session object (e.g. set initial capacity and load
on hashtables, clean up hashtables, reuse DataTables, etc.). I no
longer see a continuous climb in memory usage by the application.
More importantly, our % time in GC no longer hits 40+% at times
(verified through PERFMON).

I don't know if that was "the" problem, but so far today I have not
been able to get the application to hang (used to hang after about
10-15 minutes). Of course, that can all change tomorrow after the
build is done. :-)

Thanks again, Pat, for looking at that log.

Ken
 
Ken,

If your server is crashing or hanging you might want to
download our ReJuvin8 tool. It's free for the first
15-days. It can keep your site running despite problems.

It's designed to detect problems then restart or reboot.
It can restart ANY programs, services, or processes in
precise order. It doesn't fix bugs but it does give you some
breathing room while you are finding them. ReJuvin8 also
gives you email and telephone alert messages and graphic
status reports.

It can also test, alert, and report on URLs running anywhere
on any server on your net connection. So if you use multiple
servers you can use them to watch each other, too.

ReJuvin8 has both an automatic and a custom mode. In the
auto mode you make all the settings by pointing and
clicking. This will handle most applications.

The custom mode uses heartbeat synch files to get absolute
detection that key processes are working. Tools and
instructions are provided to help you do this. We also offer
customization as a contract service.

Charles


Get ReJuvin8 at: http://www.rejuvin8.net?ac=ch1
 
Back
Top