Unable to make the session state request to the session state server.

[dm1608]

For about 4-5 months now, I seem to be having this sporadic issue--mainly
during our busiest time of the day between 10:30-11:45AM, where all my
Windows 2003 web servers in a Microsoft NLB cluster start throwing session
state server errors. A sample error is below.

System.Web.HttpException: Unable to make the session state request to the
session state server. Please ensure that the ASP.NET State service is
started and that the client and server ports are the same. If the server is
on a remote machine, please ensure that it accepts remote requests by
checking the value of
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aspnet_state\Parameters\AllowRemoteConnection.
If the server is on the local machine, and if the before mentioned registry
value does not exist or is set to 0, then the state server connection string
must use either 'localhost' or '127.0.0.1' as the server name. at
System.Web.SessionState.OutOfProcSessionStateStore.MakeRequest(StateProtocolVerb
verb, String id, StateProtocolExclusive exclusiveAccess, Int32 extraFlags,
Int32 timeout, Int32 lockCookie, Byte[] buf, Int32 cb, Int32 networkTimeout,
SessionNDMakeRequestResults& results) at
System.Web.SessionState.OutOfProcSessionStateStore.SetAndReleaseItemExclusive(HttpContext
context, String id, SessionStateStoreData item, Object lockId, Boolean
newItem) at
System.Web.SessionState.SessionStateModule.OnReleaseState(Object source,
EventArgs eventArgs) at
System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean&
completedSynchronously)Now I'm using ASP.NET State service on a centralized
back-end Windows 2003 server that all servers communicate to.

I was originally using SQL Server state for a couple years as well prior to
having this issue. The problem with SQL wqas that when the issue occurred,
it created a blocking situation which essentially impacted all users across
all servers. The product company recommended that I use the standard ASP.NET
State service as that was what they technically supported. Why this would
make a difference is beyond me -- but I had no choice but to try it!

I have attempted to create multiple application pools, adding additional
servers, chaning TCP/IP timeout from 20 to 30 seconds, and even calling
Microsoft ASP.NET product support, with very little success. I even
recommended that they review whether they are using read-only session state
instead of read/write per page request -- as I understand that this
basically causes every page to make round-trips to state server even if
state isn't being used on the page.

Unfortunately, the application is developed by our product company and they
insist that it is something with my environment because other clients do not
have these sort of issues. However, I've talked to other clients and they
tell me when they've seen issues like they, they've basically had to create
another web farm. This issue almost seems like I've simply reached some
architectural limit within the application...

Microsoft's position on the issue is that the session state needs to be
reduced and the returncode being reported back from the state server
indicates buffers are full.

To better understand the scope of issues (rather than wait for customers to
call and complain), I installed ELMAH and configured it to send me e-mails
when unhandled exceptions occur. I basically get 500-1000 e-mails during the
time period of high activity!

If any one has any other ideas I could try or better ways to troubleshoot,
I'd appreciate it.

 

[Tom Dacon]

If any one has any other ideas I could try or better ways to troubleshoot,
I'd appreciate it.

You don't say how many concurrent sessions you're seeing when this happens
or how much state information is being held in session state for each of the
sessions, so it's hard to say whether you're just maxing out the capacity of
the staging server. A little back-of-the-envelope work on that might help
you to see if that's the issue. If it so, then of course you're going to
have to scale for the peak periods, either scale up or scale out. No
surprises there.

At my company, we've begun using a third-party solution for session store,
called NCache from Alachisoft (www.alachisoft.com). It's primarily designed
for web site caching, but also works well as a session store server across
load-balanced application servers. It's easy to scale up in a variety of
ways, and might be worth considering. You can download and try out a
full-function evaluation at no cost. I have no relationship with the company
other than as a user of their software. We're satisfied with how it's
working for us but I doubt if we're stressing it particularly heavily.

Tom Dacon
Dacon Software Consulting

 

[dm1608]

When you say concurrent connections -- what is the best way to measure this?

My state server never appears to be overloaded and CPU is frequently below
40% -- most of the activity is SQL Server running on the same box. The
ASP.NET State Server service doesn't appear to be using much CPU but is
using approx 1.5GB of memory during peak usage.

I've already scaled out by adding more servers and that doesn't appear to
provide any relief in the errors that I'm seeing.

I have no idea how large the sessions are for the various applications and I
have no access or direct access to the developers of the application. I'm
just an end-user pointing out all their flaws.

There applications work great for small shops but we're using it in a
datacenter environment where there is a lot more utilization than what a
typical end-user would experience. I'm not the only datacenter using the
product however. There is nothing special about my environment either --
just Windows 2003, NLB, and ASP.NET 2.0 running under ESX 3.5 over a SAN
back-end.

 

[bruce barker]

one issue with using state server rather than sqlserver state, is memory
usage. if there is too much, the stateserver can only throw errors. if
it gets a fatal error, it a recycle and all session data is lost.

you should be using 64bit also.

you should check the size of you session data, there is now real method,
but you can serialize to a binary buffer, and check the buffer size to
get an idea. if the data is too large you may be getting network errors,
which i suspect is the real problem. have your network people do traces
at the busy time and look for network timeouts and lost connection.

-- bruce (sqlwork.com)