System.Threading.Timer is intermittently late

[Guest]

I have an application which uses many instances of System.Threading.Timer.
This application is housed inside an HTTPAsyncHandler. When I receive certain
types of messages, I need to take an action 90 seconds later. For each
session I have a System.Threading.Timer created with infinite values for the
two intervals.. When one of these messages is received, I Change() the timer
so that the first interval is 90 seconds - the second is set to Infinite.

For the most part, the timers work just fine. However, I notice that rather
frequently (maybe 1 in 100), the timer takes much longer than 90 seconds to
fire - sometimes 100 sec, sometimes up to 150 sec. It looks like the timer
does fire - it's just late. This late firing is causing all sorts of other
problems with the app.

There does not seem to be a correlation with load - sometimes this happens
when there is only one active session, and nothing else is happening in the
meantime. Of course it only happens in production, not in development or
test. The production machine at the moment is a virtual server running under
win2003 web edition. SQL Server Express is also running on that server.

I've seen the KB article about Timers not firing in some circumstances under
load, but that appears to apply only to .NET 1.1. (I'm running 2.0).

Any suggestions as to what might be going on or how I can fix this?

 

[Peter Duniho]

[...]
For the most part, the timers work just fine. However, I notice that
rather
frequently (maybe 1 in 100), the timer takes much longer than 90 seconds
to
fire [...]

Any suggestions as to what might be going on or how I can fix this?

Nothing specific. Two things though. First, I don't see any need to
create a timer with an infinite value. Just start and stop a given timer
as needed. I would be surprised if this is related to your problem, but
you never know. Second, if you expect a useful response, you need to post
a concise-but-complete code sample that reliably reproduces the problem.

Often, in the process of distilling your code down to the bare minimum
required to reproduce the issue, you will discover what it is you're doing
in your own code that is causing the problem. At a minimum, doing so and
posting the results to the newsgroup provides context to anyone who might
be able to help.

Pete

 

[Guest]

Peter,

thanks for your reply.

I suspect you're confusing this timer with one of the other variants of
timer in the CLR. With a System.Threading.Timer, there are no start and stop
methods. You effectively turn it on or off by using the Change method.

I'd be happy to post a simple example, but there's no way - as I say, this
problem doesn't manifest itself in development or test environments, but only
in production. It's a pretty complicated app, and it only happens
intermittently. I've been trying to reduce it, but with no luck. That's why
I'm asking if others have seen similar behaviors.


--
....Mike

[...]
For the most part, the timers work just fine. However, I notice that
rather
frequently (maybe 1 in 100), the timer takes much longer than 90 seconds
to
fire [...]

Any suggestions as to what might be going on or how I can fix this?

Nothing specific. Two things though. First, I don't see any need to
create a timer with an infinite value. Just start and stop a given timer
as needed. I would be surprised if this is related to your problem, but
you never know. Second, if you expect a useful response, you need to post
a concise-but-complete code sample that reliably reproduces the problem.

Often, in the process of distilling your code down to the bare minimum
required to reproduce the issue, you will discover what it is you're doing
in your own code that is causing the problem. At a minimum, doing so and
posting the results to the newsgroup provides context to anyone who might
be able to help.

Pete

 

[Peter Duniho]

I suspect you're confusing this timer with one of the other variants of
timer in the CLR. [...]

Quite so. So I was at least correct in assuming that had nothing to do
with your problem.

That said, the rest of my reply is accurate, as difficult as it may be for
you to accomplish. There are a variety of reasons why a timer may appear
to be late, but I doubt that any of them actually involve a problem in the
Timer class itself.

The most likely reason that comes to mind is that you are using the same
thread pool used by the Timer class for other things, and that at the
moment the timer should be signaled, there are not any idle threads
available to handle the timer. In this case, the timer delegate will be
queued for execution until the thread pool has an idle thread available
for the timer. Note that the timer may not be the first one in line to
get the next idle thread.

But if you don't have a reproducible scenario, then even if someone has
seen similar behavior it may or may not be due to similar causes. The
best thing would be to identify what in your code is causing the
behavior. This can often be accomplished by gradually removing or
disabling areas of the code, until the problem goes away...even in
complicated code there is usually a way to accomplish this, resulting in
performance that wouldn't be acceptable in the real world but at least
helps diagnose the issue.

Pete

 

[Jeffrey Tan[MSFT]]

Hi Mike,

Actually, this type of production timer late issue is tough to troubleshoot
in newsgroup.

Looking at the nature of this issue, it would require intensive
troubleshooting which would be done quickly and effectively with direct
assistance from a Microsoft Support Professional through Microsoft Product
Support Services. You can contact Microsoft Product Support directly to
discuss additional support options you may have available, by contacting us
at 1-(800)936-5800 or by choosing one of the options listed at
http://www.microsoft.com/services/microsoftservices/srv_support.mspx.

Personally, I agree with Peter's reply that this late may be caused by no
free threads in thread pool for invoking the timer callback. However,
without definite diagnostic and debugging, it is hard to see this is the
real root cause. So Microsoft PSS should be a better support channel for
you.

Thanks for your understanding.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Chris Mullins [MVP]]

What's the chance that you timer isn't firing because all 25 thread pool
threads are busy doing something?

You mentioned that you "Many Instances", so it sounds as if that could be
it. If all 25 threadpool threads are busy, there's no way for the timer
(which calls you back on a threadpool thread) can do anything.

 

[Guest]

I thought that might be the case, even though I was able to catch it once
when virtually nothing was happening in the application.

I added some logging that reports ThreadPool.GetAvailableThreads. I seem to
always get 99 worker threads and 100 completion port threads available. Is
that the appropriate thing to test?

and why are the values 99, rather than 24?

Could the fact that this is running under virtual server be relevant?

 

[Chris Mullins [MVP]]

Well, the Completion Port threads are something else entirely. They're
realted to Async I/O, and they're not relevant.

As for the value of 25 vs 99, that's not really much of a clue either. The
'25' value no longer applies to the Thread Pool, as documented by Duffy at:
http://www.bluebytesoftware.com/blog/PermaLink,guid,ca22a5a8-a3c9-4ee8-9b41-667dbd7d2108.aspx

.... but if you're having minor stuttering problems, the CLR Thread Pool may
be saying, "I've got 5 threads, and I think that's enough". Then realizing
it needs another one, and creating it. I'm not sure what that delay is.

I don't think this is the case, but it's a possability.

 

[Peter Duniho]

[...]
... but if you're having minor stuttering problems, the CLR Thread Pool
may
be saying, "I've got 5 threads, and I think that's enough". Then
realizing
it needs another one, and creating it. I'm not sure what that delay is.

From reading the docs, my understanding is that the thread pool maintains
a small margin of idle threads, up to a specified maximum number of total
threads. It only checks the count of idle threads every half second, and
so I suppose if you had code that quickly started consuming threads in the
thread pool, then you could get delays even without reaching the thread
pool's maximum number, since you'd incur the cost of creating the thread
needed to service the timer event handler before the thread could be run.

How that would lead to a timer being delayed by tens of seconds, I can't
say. I wouldn't call the original problem description a "minor stuttering
problem". But you could easily get smaller delays, maybe even as long
as a second depending on what else is going on, just from not having a
thread idle and waiting to run the timer event.

(It should go without saying that on a heavily loaded system, where the
timer event handler code has been swapped out of physical RAM, all manner
of delay could occur, easily tens of seconds or longer depending on what
other i/o might be going on. I'm assuming that such a trivial explanation
has already been ruled out by the OP).

Pete

 

[William Stacey [C# MVP]]

Not sure, but I don't like changing the interval on an existing timer
object, I have seen some weird issues with that and just prefer to side step
any issue all together.
Just create a new timer as needed like so:

// Need to start some work in 2 seconds now.
System.Threading.Timer t = null;
t = new System.Threading.Timer(
delegate(object state)
{
t.Dispose();
Console.WriteLine("My stuff here after 2 sec.");
}, null, 2000, -1);

If this helps, then we found the issue. If not, then most likely hitting TP
max as others have said. That would be easy to diag a Max issue with a
simple interlocked counter var.
Also, might you be making blocking calls in the callback? If possible, you
should make no blocking calls in the callback or make sure they are very
short. If you need to block, then fire that work on another thread or a
seperate queued worker thread pool.

--
William Stacey [C# MVP]



|I have an application which uses many instances of System.Threading.Timer.
| This application is housed inside an HTTPAsyncHandler. When I receive
certain
| types of messages, I need to take an action 90 seconds later. For each
| session I have a System.Threading.Timer created with infinite values for
the
| two intervals.. When one of these messages is received, I Change() the
timer
| so that the first interval is 90 seconds - the second is set to Infinite.
|
| For the most part, the timers work just fine. However, I notice that
rather
| frequently (maybe 1 in 100), the timer takes much longer than 90 seconds
to
| fire - sometimes 100 sec, sometimes up to 150 sec. It looks like the timer
| does fire - it's just late. This late firing is causing all sorts of other
| problems with the app.
|
| There does not seem to be a correlation with load - sometimes this happens
| when there is only one active session, and nothing else is happening in
the
| meantime. Of course it only happens in production, not in development or
| test. The production machine at the moment is a virtual server running
under
| win2003 web edition. SQL Server Express is also running on that server.
|
| I've seen the KB article about Timers not firing in some circumstances
under
| load, but that appears to apply only to .NET 1.1. (I'm running 2.0).
|
| Any suggestions as to what might be going on or how I can fix this?
| --
| ...Mike

 

[Guest]

I'll try destroying and re-creating a new timer - thanks for the idea. I was
reusing the timer, since I was a little worried about extra work for the GC
et al.

But I also tried another simpler experiment: I fired off a new thread (not a
pool thread) and just wrote a simple loop with a sleep(2000), then logging a
message. This loop seems to follow the same pattern as the timer - every once
in a while, ther's a gap of 10-15 seconds! That seems to exonerate the thread
pool.

blocking in the callback is another idea - in all of these cases, I am
logging to the database, so it is conceivable that I am blocking there. But
I've been logging the durations of the database calls as well, and nothing
takes longer than a few ms. I would find it hard to believe that a DB call
was taking 10 seconds.

but I'll try doing the logging via an async call, see if that makes a
difference. thanks
--
....Mike

Not sure, but I don't like changing the interval on an existing timer
object, I have seen some weird issues with that and just prefer to side step
any issue all together.
Just create a new timer as needed like so:

// Need to start some work in 2 seconds now.
System.Threading.Timer t = null;
t = new System.Threading.Timer(
delegate(object state)
{
t.Dispose();
Console.WriteLine("My stuff here after 2 sec.");
}, null, 2000, -1);

If this helps, then we found the issue. If not, then most likely hitting TP
max as others have said. That would be easy to diag a Max issue with a
simple interlocked counter var.
Also, might you be making blocking calls in the callback? If possible, you
should make no blocking calls in the callback or make sure they are very
short. If you need to block, then fire that work on another thread or a
seperate queued worker thread pool.

--
William Stacey [C# MVP]



|I have an application which uses many instances of System.Threading.Timer.
| This application is housed inside an HTTPAsyncHandler. When I receive
certain
| types of messages, I need to take an action 90 seconds later. For each
| session I have a System.Threading.Timer created with infinite values for
the
| two intervals.. When one of these messages is received, I Change() the
timer
| so that the first interval is 90 seconds - the second is set to Infinite.
|
| For the most part, the timers work just fine. However, I notice that
rather
| frequently (maybe 1 in 100), the timer takes much longer than 90 seconds
to
| fire - sometimes 100 sec, sometimes up to 150 sec. It looks like the timer
| does fire - it's just late. This late firing is causing all sorts of other
| problems with the app.
|
| There does not seem to be a correlation with load - sometimes this happens
| when there is only one active session, and nothing else is happening in
the
| meantime. Of course it only happens in production, not in development or
| test. The production machine at the moment is a virtual server running
under
| win2003 web edition. SQL Server Express is also running on that server.
|
| I've seen the KB article about Timers not firing in some circumstances
under
| load, but that appears to apply only to .NET 1.1. (I'm running 2.0).
|
| Any suggestions as to what might be going on or how I can fix this?
| --
| ...Mike