Synchronous HttpWebRequest and HttpRebResponse

  • Thread starter Thread starter Steve Binney
  • Start date Start date
S

Steve Binney

My code makes synchronous HttpWebRequest and HttpRebResponse calls. In VS
2003, I am getting memory leaks and event handle leaks. I am closing all
streams and using "using"statements. I have used .Net memory profiler from
Sci Tech to analyze the leaks and they are coming from inside HttpWebRequest
and HttpRebResponse. When I run my code on VS2005, I do not get any leaks.
Is this problem fixed in .Net 1.1 SP1? I may not be able to wait for VS
2005.

Steve Binney

See this blog for a detailed description of the problem:
http://dturini.blogspot.com/2004/06/on-past-few-days-im-dealing-with-some.html
Damn! Handle leaks on the CLR!
On the past few days, I'm dealing with some hard to find resource leaks. We
seem to be pushing some areas CLR to their limit. You may say, "hey, how can
you get resource leaks in a garbage collected environment"?.
Well, the simple code below will leak lots of handles.

private void MakeARequest() { HttpWebRequest wr =
(HttpWebRequest)WebRequest.Create("http://www.microsoft.com/");
wr.Proxy = WebProxy.GetDefaultProxy(); wr.Proxy.Credentials =
CredentialCache.DefaultCredentials; using (HttpWebResponse wrp =
(HttpWebResponse)wr.GetResponse()) { using (Stream
s = wrp.GetResponseStream()) { using
(StreamReader sr = new StreamReader(s))
{ string st = sr.ReadToEnd();
sr.Close();

s.Close(); } } } }

Did you see it? It isn't missing a single Close() or Dispose() call, and yet
leaks lots of handles. The biggest the page, the more handles it leaks.
I've been using an evaluation version (10 days remaining...) of the great
SciTech's .NET memory profiler and noticed that the CLR is forgetting to
dispose some (a lot!) ManualResetEvent, Sockets, and some internal Streams
and Readers.
After a while, I found that the culprit are the synchronous HttpWeb(Request,
Response) methods, and, by calling the assynchronous versions, I was able to
reduce it a lot. Even the returned stream, System.Net.ConnectStream have
these problems.
The only problem is when the request throws an Exception, it still leaks
lots of handles, and, by now, it seems that I can do nothing about it...

Steve Binney
 
Steve,

Are you sure that the handles never get collected ? Can you put a call
to GC.Collect() per 'x' calls of httpwebrequest (eg: x=1000) and see
if the leaks go away ?

Since CLR has a GC, GC collection wont happen unless the system feels
memory pressure. It is feasible that the resources get cleaned up at
the next GC, and in your test runs, the GC is not happening.

If GC.Collect() doesnt end up closing the handles, then it could very
well be a handle leak.


feroze
===========================
This posting is provided as-is. It provides no guarantees and confers
no rights.
 
Feroze,
I have run with the released (not Sp1) version of .Net 1.1 and with VS2005
Beta 1. I am running for approx 1 hour and I am making two web
requests/responses per second.. This is what I observe:

..Net 1.1 Released (v1.1.4322)
The handle count grows to between 1000 and 2000 and eventually are garbage
collected. If I call gc.Collect in my code, the handles do not grow. Using
Reflector, I can seet that an event is allocated on every call to
HttpWebRequest.GetRequestStream and HttpWebRequest.GetResponse and is never
closed. Additionally, I am seeing in process explorer a significant memory
leak. The profiler is showing leaking in System.String. The memory leak
continues to grow and the memory is never recovered even if GC.Collect is
called.

VS 2005 SP1 (.Net 2.0)
Using the same code, I saw no memory or handle leaks. With Reflector I was
that HttpWebRequest.GetRequestStream and HttpWebRequest.GetResponse close
all event handles.

Since this was fixed in VS2005 SP1, I was wondering if this was fixed in
..Net 1.1 SP1.

Steve
 
Hi, Feroze, as I've posted some very simple test code on
http://dturini.blogspot.com/2004/06/on-past-few-days-im-dealing-with-some.html
I've been dealing with these leaks. The leaks are huge, and calling
GC.Collect() (a bad practice, btw, but it helps debugging) directly
from my code cleans up handles, but memory keeps raising.
Just put the code I show on multiple (e.g. 30) threads, call it, say
1000 times on each thread and you'll understand what I mean: you'll
have an application with around 500MB RAM allocated an almost 1500
handles. At some point, the application simply freezes up. I have an
application, that when under stress can achieve this kind of load and
httpwebrequest simply freezes up the application after a while.
 
Ok, I will follow up with our dev team and see what they have to say.

===========================
This posting is provided as-is. It provides no guarantees and confers no
rights.

Daniel Turini said:
Hi, Feroze, as I've posted some very simple test code on
http://dturini.blogspot.com/2004/06/on-past-few-days-im-dealing-with-some.html
I've been dealing with these leaks. The leaks are huge, and calling
GC.Collect() (a bad practice, btw, but it helps debugging) directly
from my code cleans up handles, but memory keeps raising.
Just put the code I show on multiple (e.g. 30) threads, call it, say
1000 times on each thread and you'll understand what I mean: you'll
have an application with around 500MB RAM allocated an almost 1500
handles. At some point, the application simply freezes up. I have an
application, that when under stress can achieve this kind of load and
httpwebrequest simply freezes up the application after a while.

(e-mail address removed) (Feroze [MSFT]) wrote in message
Steve,

Are you sure that the handles never get collected ? Can you put a call
to GC.Collect() per 'x' calls of httpwebrequest (eg: x=1000) and see
if the leaks go away ?

Since CLR has a GC, GC collection wont happen unless the system feels
memory pressure. It is feasible that the resources get cleaned up at
the next GC, and in your test runs, the GC is not happening.

If GC.Collect() doesnt end up closing the handles, then it could very
well be a handle leak.


feroze
===========================
This posting is provided as-is. It provides no guarantees and confers
no rights.


"Steve Binney" <[email protected]> wrote in message
Click to expand...
Click to expand...
 
Can you do one more thing ? Instead of just doing a "GC.Collect()" after
every 'x' requests, do:

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();

And see if this ends up collecting all objects ?

feroze
=========================
This posting is provided as is. It provides no guarantees and confers no
rights

=========================

Daniel Turini said:
Hi, Feroze, as I've posted some very simple test code on
http://dturini.blogspot.com/2004/06/on-past-few-days-im-dealing-with-some.html
I've been dealing with these leaks. The leaks are huge, and calling
GC.Collect() (a bad practice, btw, but it helps debugging) directly
from my code cleans up handles, but memory keeps raising.
Just put the code I show on multiple (e.g. 30) threads, call it, say
1000 times on each thread and you'll understand what I mean: you'll
have an application with around 500MB RAM allocated an almost 1500
handles. At some point, the application simply freezes up. I have an
application, that when under stress can achieve this kind of load and
httpwebrequest simply freezes up the application after a while.

(e-mail address removed) (Feroze [MSFT]) wrote in message
Steve,

Are you sure that the handles never get collected ? Can you put a call
to GC.Collect() per 'x' calls of httpwebrequest (eg: x=1000) and see
if the leaks go away ?

Since CLR has a GC, GC collection wont happen unless the system feels
memory pressure. It is feasible that the resources get cleaned up at
the next GC, and in your test runs, the GC is not happening.

If GC.Collect() doesnt end up closing the handles, then it could very
well be a handle leak.


feroze
===========================
This posting is provided as-is. It provides no guarantees and confers
no rights.


"Steve Binney" <[email protected]> wrote in message
Click to expand...
Click to expand...
 
Hi Daniel,

I did try your repro code with 30 threads (See attached code, which I did
tried) for 15 minutes, can't see any indication of memory leak. I did try
with .Net framework version 1.1. Can you tell me little detail about your
network configuration, what proxy you are hitting, what OS you are trying.
Also tell me if this code really represent the scenario you are describing
for leak.

Thanks

Adarsh





Daniel Turini said:
Hi, Feroze, as I've posted some very simple test code on
http://dturini.blogspot.com/2004/06/on-past-few-days-im-dealing-with-some.html
I've been dealing with these leaks. The leaks are huge, and calling
GC.Collect() (a bad practice, btw, but it helps debugging) directly
from my code cleans up handles, but memory keeps raising.
Just put the code I show on multiple (e.g. 30) threads, call it, say
1000 times on each thread and you'll understand what I mean: you'll
have an application with around 500MB RAM allocated an almost 1500
handles. At some point, the application simply freezes up. I have an
application, that when under stress can achieve this kind of load and
httpwebrequest simply freezes up the application after a while.

(e-mail address removed) (Feroze [MSFT]) wrote in message
Steve,

Are you sure that the handles never get collected ? Can you put a call
to GC.Collect() per 'x' calls of httpwebrequest (eg: x=1000) and see
if the leaks go away ?

Since CLR has a GC, GC collection wont happen unless the system feels
memory pressure. It is feasible that the resources get cleaned up at
the next GC, and in your test runs, the GC is not happening.

If GC.Collect() doesnt end up closing the handles, then it could very
well be a handle leak.


feroze
===========================
This posting is provided as-is. It provides no guarantees and confers
no rights.


"Steve Binney" <[email protected]> wrote in message
Click to expand...
Click to expand...
 
I've got the same problem using a webservice returning datasets. The
dataset, tables, and colums are reported to be undisposed. Is there a
fix for this? I've tried the suggestions posted but to no avail.
/Patrik
 
Back
Top