w3wp high cpu, garbage collection, lots of objects stored in Sessi

  • Thread starter Thread starter Larry Epn
  • Start date Start date
L

Larry Epn

I have a web-based application that tries its darndest to work like a windows
form app. It uses lots of objects to retain state across multiple forms, and
all of these complex objects are stored in many session variables whose names
include GUIDs since there are so many of them. In other words, it's a MEMORY
PIG - but it gives the functionality/experience necessary. Also, users tend
to stay in this application for an entire work shift (8-10 hours) and the
session is kept alive via a keep-alive ajax-based timer panel that guarantees
that even if there is no user activity that this keeps the session alive.
All of these session objects are programmatically assigned a timeout duration
at which time they are disposed during a common routine that occasionally
gets triggered.
The problem arises usually late in the day when the memory utilization of
the w3wp.exe process begins banging against that 99% CPU usage, and it stays
there for an eternity. Amazingly, the application continues to function,
albeit noticeably slower. I have gone through with a fine-tooth comb to find
places where I am not disposing or releasing references to these complex
session objects. As much as I may like to think, I obviously haven't found
them all (???)
Is there a way to see specifically why the garbage collector (I ASSUME) is
looping around this memory in a seemingly infinite loop, and then why it
never seems to stop unless I reset the process (and LOSE EVERYONE's SESSION
variables which basically crashes their application - not a good workaround).
I have even attempted to use other session management schemes (SQL and the
session state service), but some of these objects just won't serialize even
though I've attempted to make them serializable (another topic, I guess).
Has anyone else run into this situation? Any ideas?
Thank you,
Larry
 
Hi Larry,

My name is Allen Chen. It's my pleasure to work with you on this issue.

From your description the CPU usage is 99%. As you said, this issue is
generally caused by infinite loop. However for now we cannot assume if it's
caused by GC. Normally to troubleshoot high CPU issue we'll use Debug
Diagnostics 1.1 to create a hang dump and generate a report for us (you can
refer to the help documentation of this tool to learn how to use it):

http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1286

From the report we can see which thread occupies the most CPU time. In
addition other clues will be included in the report.

Then we can use WinDbg to analyze the dump to see what the root cause is.

http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx

Though dump analysis is out of the support boundary of managed newsgroup
you can send the report to me:

(e-mail address removed)

I'll do my best to see if I could provide any help on this anyway.

Reference FYI:

http://blogs.technet.com/marcelofartura/archive/2006/09/15/troubleshooting-i
is-100-cpu-issues-step-by-step-intermediary.aspx

http://blogs.msdn.com/tom/archive/2008/05/02/asp-net-tips-what-to-gather-to-
troubleshoot-part-1-high-cpu.aspx

Regards,
Allen Chen
Microsoft Online Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
 
Hi Larry,

Do you have any progress on this issue?

Regards,
Allen Chen
Microsoft Online Support
 
I had downloaded/installed the dump utility and created a hang dump and a
report.
I don't really know what I'm looking for. I truly appreciate your
willingness to have a view of it and give me some direction.
I have sent you an e-mail with a link to the file. It is a very large zip
file > 120MB.
Regards,
Larry
 
Hi Larry,

Thanks for your update.

Unfortunately dump analysis is out of our support boundary. See below:

http://blogs.msdn.com/msdnts/archive/2006/11/08/msdn-service-introduction.as
px

So I could not help you analysis the dump file. To analyze dump file you
can contact CSS for incident based support:
http://support.microsoft.com/select/default.aspx?target=assistance&ln=en-us

Our dedicated support engineer will work with you to investigate this issue.

Let's focus on what assistance I can provide. I got something from the
report. To get the report please try following steps:

1. Launch Debug Diagnostic Tool.
2. Click "Advanced Analysis" tab. Make sure "Crash/Hang Analyzers" is
selected.
3. Click "Add Data Files" button, select the dump file and click "Open".
4. Click "Start Analysis" button.
5. A report will be opened when the analysis is done.

In the report, we first check the top 5 threads by CPU time. We can see the
thread 19, 20, 21, 51 seem abnormal. Each of these four threads take more
than 2:30:00 CPU time on average.

Look into the call stack of these four threads (it cannot be seen in the
report. I grabbed info from dump. But I'm afraid I cannot provide further
assistance on the dump file). I found this method is included in all of the
call stack.

App_Code.CommonStatic.CleanoutOldSessionObjects(System.Web.SessionState.Http
SessionState)

So I think it may be the root cause. If you can provide the implementation
of this method I can help to check if it has the potential to cause 100%
CPU issue. If it's not caused by this method probably you have to contact
CSS to analyze the dump file.


Regards,
Allen Chen
Microsoft Online Support
 
Forgot to mention that you can remove that method to see if CPU usage is
still high. In this way we can know if it's caused by it.

Regards,
Allen Chen
Microsoft Online Support
 
Thanks for the help. Your indication of the method name may be all I need, I
hope.
The method you have indicated is a method that I had written. It goes
through and expires old session objects based upon a timestamp property in
each object that indicates that logically the object can be removed from the
user's session. Simply it means that "it is not likely that the user will
hit the *BACK* button and recreate the page tied to this session object since
the page would be over 1/2-hour old, so go ahead and clean out its session
objects."
Obviously, I have a coding issue here, and I am happy to know that it is in
my control and not something in the OS, IIS or ASP.NET environments.
 
You're welcome Larry. I think you can comment this method first to test. If
CPU usage is normal after that we can assert this is the root cause.

Please keep me updated.

Regards,
Allen Chen
Microsoft Online Support
 
Hi Larry,

Do you have any progress on it?

Regards,
Allen Chen
Microsoft Online Support
 
Back
Top