Memory usage anomaly with asyncrhonous server application

  • Thread starter Thread starter jack.mott
  • Start date Start date
J

jack.mott

I am having a strange memory issue in my server application. This
application accepts socket connections, passes messages around, then
closes them.

Using the .NET memory profiler from scitech.com, it reports reasonable
memory behavior, hovering around 400k for live bytes, and peaking no
higher than 1.5megs or so for total bytes used.

However, looking in the task manager at the process, it reports the
memory being used at around 50megs, climbing about 1 meg every second.
It will get as high as 700megs+!!

Are there some known memory leaks in .net?
Could I be doing something wrong in my C# code to cause this?
 
I have done simmilar thing - there were no memory leaks.
Just a few pointers:
a) make sure you dereference old buffers once you finished processing
b) To make sure things arent getting cached try running garbage collection
after each loop/deallocation - check the gargbage collection statistics from
performance monitor - see if something is actually happenning
c) When you look at task manager make sure you look at the process memory
utilization and not total memory consumption windows has a wierd way of
caching things. One way i found to flush the cache (file or otherwise) is to
write a simple program that allocates 90% of your ram and then frees it (just
make sure you write it in C / C++)

Other then that - post the code ;)
 
Like I said, the .net memory profiler shows the garbage collection
doing what it should. Memory there is constant. Looking in task manage
specifically at my process though shows things constantly growing.

I did get some more info from this profiling too. It seems to be that
the native memory growth is occuring in what the profiler calls "holes"

private-->managed heap-->normal heap-->generation #2--->"holes"

according to the proflier holes are:

"Holes" represent memory that is unused between two allocated
instances. "Holes" appear when the heap is not fully compacted, due
to pinned instances or optimizations in the garbage collector.


anyone have any idea what it is talking about? what is a pinned
instance?
 
jack said:
Like I said, the .net memory profiler shows the garbage collection
doing what it should. Memory there is constant. Looking in task manage
specifically at my process though shows things constantly growing.

I did get some more info from this profiling too. It seems to be that
the native memory growth is occuring in what the profiler calls "holes"

private-->managed heap-->normal heap-->generation #2--->"holes"

according to the proflier holes are:

"Holes" represent memory that is unused between two allocated
instances. "Holes" appear when the heap is not fully compacted, due
to pinned instances or optimizations in the garbage collector.


anyone have any idea what it is talking about? what is a pinned
instance?
Click to expand...

Not sure if the profiles makes a distinction between Gen2 and the Large
Object heap (for objects > 85Kb), but could you check the sizes of both
using Perfmon?
The reason I ask is that, unlike the Gen0, 1 and 2 heaps, the LOH isn't
compacted, the result is that LOH gets fragmented over time depending on
your memory allocation scheme, if you allocate random sized objects in a
random fashion this isn't an issue, but if your allocate objects that tend
to grow in time, its possible to end with such a large number of holes that
cannot be used to store the ever growing objects, that no memory can be
allocated any further, and the process dies with a "Out of memory"
exception.
So the questions you have to answer are:
- Is the LOH growing?
- Are you allocating large objects (normally arrays > 85 Kb).
- Is you allocation scheme random or not.

Willy.
 
Ok yes it distinguishes between the normal heap and the large object
heap.

No the large object heap is not growing.

No I am not allocating large objects.

Allocation scheme random? Can you elaborate? If the allocation scheme
is too "regular" what sort of techniques do people use to "randomize"
it? Sounds like an odd consideration.
 
I reread your message and now understand the potential problem if
allocation continues to grow (can't fill in the holes!)

There is one use case where the allocation tends to grow (small objects
though), but as sessions end they should all get collected in one big
chunk. I can play with this though and just allocate a large chunk to
begin with and see what that does.
 
jack said:
I reread your message and now understand the potential problem if
allocation continues to grow (can't fill in the holes!)

There is one use case where the allocation tends to grow (small objects
though), but as sessions end they should all get collected in one big
chunk. I can play with this though and just allocate a large chunk to
begin with and see what that does.
Click to expand...

No don't waste your time with this, this only applies to the LOH, the other
heaps are compacted by the GC, except when a pinned object prevents full
compactation.
So I guess the "holes" the profiler sees are a result of pinning the buffers
for the duration of the reads/writes from/to a socket, nothing to worry
about.

If I recall, the GC heaps in total do not exceed 1.5 Mb, right? so it looks
like you have a unmanaged memory leak.
Can you check the "private bytes" counter for the process? Do you call into
unmanaged code or COM, wheter directly or indirectly?

Willy.
 
Thing is these "holes" it is seeing are huge, like 20 megabytes worth
of holes. There isn't that much socket data being passed around.


As for calling into unmanaged code, not directly but I imagine all of
the socket calls do this indirectly. Perhaps I am not properly
releaseing something with the sockets after reads/writes or
disconnects?
 
jack said:
Thing is these "holes" it is seeing are huge, like 20 megabytes worth
of holes. There isn't that much socket data being passed around.


As for calling into unmanaged code, not directly but I imagine all of
the socket calls do this indirectly. Perhaps I am not properly
releaseing something with the sockets after reads/writes or
disconnects?
Click to expand...

Not sure I'm following here, you said managed memory tops at 1.5 Mb, but the
profiler says 20Mb "holes", normally managed memory should include all real
object data and the holes.
I suggest you watch the CLR memory counters with perfmon to be sure.

Willy.
 
I'm sorry for perhaps being a bit ignorant of the terminology. But the
managed memory as reported in the 'native memory' area of the profiler
gets quite huge. When I look at the total size of all the C# objects,
that is what tops out at 1.5Mb
 
Here is some of the relevant networking code, SocketClosed() calls
socket.Close()
Anything I am doing wrong here that might be causing native memory
leaks?


public void OnClientConnect(IAsyncResult asyn)
{

Socket socket=null;
try
{
socket = m_socListener.EndAccept (asyn);
socket.SetSocketOption (SocketOptionLevel.Socket,
SocketOptionName.KeepAlive, 1);
m_socListener.BeginAccept(new AsyncCallback ( OnClientConnect
),null);
socketList.Add(socket);
updateSocketList();
textNumConnections.Text=""+socketList.Count;
WaitForData(socket,null);
}
catch(ObjectDisposedException ode)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("ODE Exception follows:");
listBoxTraffic.Items.Add(ode);
}
SocketClosed(socket);
}
catch(SocketException se)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("Socket Exception Follows:");
listBoxTraffic.Items.Add(se);
}
SocketClosed(socket);
}

}


public void WaitForData(System.Net.Sockets.Socket soc, MsgState
msgState)
{
lock(this)
{
try
{
if (msgState == null)
{
msgState = new MsgState (soc);
}

// now start to listen for any data...

soc.BeginReceive
(msgState.dataBuffer,msgState.pos,msgState.expectedSize-msgState.pos,SocketFlags.None,new
AsyncCallback (OnDataReceived),msgState);

}
catch (ObjectDisposedException ode)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("ODE Exception follows:");
listBoxTraffic.Items.Add(ode);
}
SocketClosed(soc);
}
catch(SocketException se)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("Socket Exception Follows");
listBoxTraffic.Items.Add(se);
}
SocketClosed(soc);
}
}

}

public void OnDataReceived(IAsyncResult asyn)
{
lock (this)
{
MsgState msgState=null;
try
{

msgState = (MsgState)asyn.AsyncState ;

int bytesReceived = 0 ;
bytesReceived = msgState.socket.EndReceive (asyn);
msgState.pos += bytesReceived;

//A result of 0 implies that the socket has died
if (bytesReceived == 0)
{
SocketClosed(msgState.socket);
return;
}

//Check the type of message, handle it, then wait for the next
chunk of data
msgState = processMessage(msgState);
if (msgState != null)
{
WaitForData(msgState.socket,msgState);
}
}
catch (ObjectDisposedException ode)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("ODE Exception follows:");
listBoxTraffic.Items.Add(ode);
}
this.SocketClosed(msgState.socket);
}
catch(SocketException se)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("Socket Exception follows:");
listBoxTraffic.Items.Add(se);
listBoxTraffic.Items.Add(se.ErrorCode);
listBoxTraffic.Items.Add("msg pos:"+msgState.pos+" msg
exp:"+msgState.expectedSize+" socket:"+msgState.socket);
}
this.SocketClosed(msgState.socket);
}
}
}


protected void SendData(Socket socket, byte[] data, int start, int
end)
{


try
{
socket.BeginSend(data, start, end,0,
new AsyncCallback(DataSent), socket);
}
catch (ObjectDisposedException ode)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("ODE Exception follows:");
listBoxTraffic.Items.Add(ode);
}
this.SocketClosed(socket);
}
catch(SocketException se)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("Socket Exception follows from send:");
listBoxTraffic.Items.Add(se);
listBoxTraffic.Items.Add(se.ErrorCode);
}
this.SocketClosed(socket);
}
}

protected void DataSent(IAsyncResult asyn)
{
Socket socket=null;
try {

socket = (Socket)asyn.AsyncState;
socket.EndSend(asyn);
}
catch (ObjectDisposedException ode)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("ODE Exception follows:");
listBoxTraffic.Items.Add(ode);
}
this.SocketClosed(socket);
}
catch(SocketException se)
{
if (!PRODUCTION)
{
listBoxTraffic.Items.Add("Socket Exception follows:");
listBoxTraffic.Items.Add(se);
listBoxTraffic.Items.Add(se.ErrorCode);
}
this.SocketClosed(socket);
}

}
 
jack said:
I'm sorry for perhaps being a bit ignorant of the terminology. But the
managed memory as reported in the 'native memory' area of the profiler
gets quite huge. When I look at the total size of all the C# objects,
that is what tops out at 1.5Mb
Click to expand...

Jack,

Sure, the terminology can be confusing, therefore I suggest you take a look
at the perfmon counters, just to make sure we are talking about the same
"memory".
The counters to look at are the .NET CLR-Memory counters gen1, gen2 and
Large Object heap sizes for your managed process.
Other counters are the Process counters Private Bytes and Working Set for
the same process.

The Managed heap(s) are part of the Process native heaps (private bytes), if
private bytes grows without any growth of the managed heaps, it indicates a
native leak.
If the managed heap grows, it's an indication that you keep objects alive.

Willy.
 
Ok, according to perfmon, my gen1 heap goes up and down but remains
constant and resonbly small on average more or less.

the gen2 heap rises right along with the private bytes.

So this implies, according to you, that I am keeping objects alive?

Despite the .net profiler saying I am not....aggravating! =)
 
jack said:
Ok, according to perfmon, my gen1 heap goes up and down but remains
constant and resonbly small on average more or less.

the gen2 heap rises right along with the private bytes.

So this implies, according to you, that I am keeping objects alive?

Despite the .net profiler saying I am not....aggravating! =)
Click to expand...

Jack,

The Gen2 part of the managed heap is where surviving objects finaly end
after a number of Gen0/1 collections. Gen2 gets collected when a full
collect occurs, you can check the number of full collects by looking at the
#Gen2 collections counter and you can force it by calling GC.Collect();
However, if the size of gen2 doesn't drop after a Gen2 Collect, it clearly
indicates that object references are being held alive.

Willy.
 
ooh, thats interesting. Playing around on the server, I opened up some
other applications that used significant amounts of memory, then all of
a sudden my servers memory usage dropped from 700megs to 180megs!
 
jack said:
ooh, thats interesting. Playing around on the server, I opened up some
other applications that used significant amounts of memory, then all of
a sudden my servers memory usage dropped from 700megs to 180megs!
Click to expand...

Jack,

Don't pay to much attention to this, your problem is that you are consuming
too much memory.
Say you have 1GB RAM and one application's (AP1) working set is 700MB, when
you start a second application (AP2) that takes another 500MB, the Memory
manager will start trimming the AP1 WS until there is anough free memory to
accomodate AP2 WS. Note that WS trimming means reducing the number of pages
in RAM, the freed (data) pages are moved to the paging file and will be
moved back to memory when needed (hard page faults).
How much RAM do you have installed?

Willy.
 
Ah, so task manager is only reporting bytes in physical memory and not
things moved to the hd?


There is 1gig of RAM on the machine, the scenario you describe
definitely makes sense.
 
jack said:
Ah, so task manager is only reporting bytes in physical memory and not
things moved to the hd?


There is 1gig of RAM on the machine, the scenario you describe
definitely makes sense.
Click to expand...

To be precise Taskman's "Mem Usage" is the process Working Set, and the WS
is the number of bytes actualy mapped to pages in RAM.

Willy.
 
Willy if you could take a look at my new post "possible bug in dotnet
sockets". This problem is bizarre and im leaning towards thinking it
is a bug in dotnet and not my fault. I've posted sample code that
reproduces the problem.
 
Back
Top