Memory allocation limits

[Martin Plechsmid]

Hello.

When trying to allocate a huge amount of memory, our production system
throws OutOfMemoryException. Well, there is not enough memory available,
that's the reason. But what I don't understand is the folowing:

Using System.Diagnostics.PerformanceCounter("Memory", "Available MBytes") we
see that the system still has over 600 MB available. However, when the
system tries to allocate about 300 MB byte[], it gets OutOfMemoryException.
I would expect, that 700MB of available memory means that the system is able
to allocate almost all of these 700 MB in one continuous block. And if it is
not able right now, it will be able after garbage collection finishes.

It seems to me that the garbage collector needs twice as much memory for its
work than it allows to allocate. Is it so?

Thank you,
Martin.

 

[Peter Duniho]

Hello.

When trying to allocate a huge amount of memory, our production system
throws OutOfMemoryException. Well, there is not enough memory available,
that's the reason. But what I don't understand is the folowing:

Using System.Diagnostics.PerformanceCounter("Memory", "Available MBytes") we
see that the system still has over 600 MB available. However, when the
system tries to allocate about 300 MB byte[], it gets OutOfMemoryException.
I would expect, that 700MB of available memory means that the system is able
to allocate almost all of these 700 MB in one continuous block.

That's an incorrect expectation.

And if it is
not able right now, it will be able after garbage collection finishes.

It seems to me that the garbage collector needs twice as much memory for its
work than it allows to allocate. Is it so?

No. But it's not nearly as simple as you seem to think it is either.

First, "available MBytes" refers to physical RAM, which has practically
nothing to do with how much memory your process can allocate. What's
important to your own process is _virtual_ address space, and available
disk space.

Within your process's virtual address space, you need a _contiguous_
block of address space large enough to contain your allocation. In
addition, on the hard disk (where the virtual memory manager's swap file
resides) you need enough (not necessarily contiguous) space to contain
your allocation.

Most likely in your case, you are running out of virtual address space.
You may in fact even have more than 300MB of virtual address space
free, but due to fragmentation, no single contiguous block available
that large.

The fact is, 300MB is a pretty large data structure for a 32-bit
application. Except under the most ideal circumstances, I would expect
an allocation that large to succeed once, rarely twice, at the most.
Even under the best circumstances, you're not likely to get more than
four successful allocations that size, and at that point, very little
else will work.

Pete

 

[Wilson, Phil]

This might be a checkerboading scenario. You're asking for one chunk of
300MB, but that won't work if the 600MB available is in (say) 600 chunks of
1MB. 700MB available does not mean it's all in one huge available chunk.

 

[Martin Plechsmid]

This might be a checkerboading scenario. You're asking for one chunk of

300MB, but that won't work if the 600MB available is in (say) 600 chunks
of 1MB. 700MB available does not mean it's all in one huge available
chunk.

Yes, of course. But after garbage collection all separated chunks should
join in one contiguous area, shouldn't they? Some GC algorithms work this
way, but I don't know whether .NET is this case.

Martin.

 

[Martin Plechsmid]

First, "available MBytes" refers to physical RAM, which has practically

nothing to do with how much memory your process can allocate. What's
important to your own process is _virtual_ address space, and available
disk space.

I expect that the free virtual address space always contains free RAM. Thus
700MB of available RAM should mean at least 700MB of available virtual
address space - correct?

Within your process's virtual address space, you need a _contiguous_ block
of address space large enough to contain your allocation.

Yes, I know that. But I expect that garbage collection joins separated
pieces of memory into one contiguous area. Maybe this is the incorrect
expectation?

Most likely in your case, you are running out of virtual address space.
You may in fact even have more than 300MB of virtual address space free,
but due to fragmentation, no single contiguous block available that large.

The production server has 8GB of physical RAM, and it is not used to its
limits. I don't think that lack of memory resources is the cause, but I'll
check it again.

The fact is, 300MB is a pretty large data structure for a 32-bit
application. Except under the most ideal circumstances, I would expect an
allocation that large to succeed once, rarely twice, at the most. Even
under the best circumstances, you're not likely to get more than four
successful allocations that size, and at that point, very little else will
work.

According to our logs, at most one such huge structure is created at a time,
but even then it has very short life. I.e. minutes before the structure is
allocated again, and seconds before it's given available for disposal.

Thank you,
Martin.

 

[Peter Duniho]

I expect that the free virtual address space always contains free RAM. Thus
700MB of available RAM should mean at least 700MB of available virtual
address space - correct?

No. Not even close to correct. A process's virtual address space has
nothing to do with the amount of physical RAM available. There can be
no physical RAM free, and yet plenty of virtual address space available
to a process, and vice a versa.

Remember: physical RAM is a global resource, shared by all processes
running on the computer, where that sharing is managed by the virtual
memory manager. But to each process, the only thing they see is the
_virtual_ address space, which is completely independent of the physical
RAM. That's why it's called "_virtual_".

Yes, I know that. But I expect that garbage collection joins separated
pieces of memory into one contiguous area. Maybe this is the incorrect
expectation?

Firstly: for objects in the small object heap, garbage collection
_sometimes_ will defragment the heap, by compacting it. But the large
object heap is never compacted, and thus can become fragmented.

Secondly: the .NET heap compaction strategy has very little to do with
compaction of the used virtual address space itself. The .NET heap is
allocated from the virtual address space, but is managed separately from
that. The .NET heap can be perfectly compacted, and yet your process
can still have fragmentation in the virtual address space.

The production server has 8GB of physical RAM, and it is not used to its
limits. I don't think that lack of memory resources is the cause, but I'll
check it again.

The amount of physical RAM is irrelevant. The amount of disk space is
barely relevant. From your description, everything suggests that it's
the process's virtual address space that is getting fragmented, and no
amount of free physical RAM or disk space will help that.

According to our logs, at most one such huge structure is created at a time,
but even then it has very short life. I.e. minutes before the structure is
allocated again, and seconds before it's given available for disposal.

I think you misunderstand me: I mean "at most" during the entire
lifetime of the process. If you're lucky, you won't have any new
allocations in the virtual address space between the time that the large
data structure is allocated and when it's freed. But most applications
don't work that way, and it would be very easy for a single allocation
of a single structure of that size to result in enough subsequent
fragmentation to prevent future allocations of that size.

So far, you've said nothing at all about why you're allocating something
this large, or why you're trying to do so in a 32-bit process. So all I
can tell you is that the best answer to your problem is "don't do that".
Either change your application so that it's a 64-bit application, or
change your algorithm so you don't need a single 300MB data structure.

And in any case, you need to rid yourself of your incorrect assumptions
about how memory management works. They are leading you down fruitless
paths as you search for a solution to your problem.

Pete

 

[Martin Plechsmid]

I expect that the free virtual address space always contains free RAM.

No. Not even close to correct. A process's virtual address space has
nothing to do with the amount of physical RAM available. There can be no
physical RAM free, and yet plenty of virtual address space available to a
process, and vice a versa.

Aha! I thought that the PerformanceCounter was giving me information about
the memory of the .NET's GC heap (which is a part of the process memory)!
Not an info about the system memory!

And I thought that the GC pre-allocates a _contiguous_ block of memory from
the system virtual memory, which it subsequently lends to managed objects.
This was my other misunderstanding. I thought that the performance counter
shows the amount of this pre-allocated memory available for .NET heap (thus
part of the process memory).

Firstly: for objects in the small object heap, garbage collection
_sometimes_ will defragment the heap, by compacting it. But the large
object heap is never compacted, and thus can become fragmented.

Some GC algorithms (e.g. the basic copying GC) always defragment the heap.
But I do not know which algorithm is used by .NET. And if - as you say - the
heap memory need not be contiguous, this can influence its fragmentation
too. I believe you are right that the .NET GC algorithm need not defragment
the heap.

So far, you've said nothing at all about why you're allocating something
this large, or why you're trying to do so in a 32-bit process. So all I
can tell you is that the best answer to your problem is "don't do that".
Either change your application so that it's a 64-bit application, or
change your algorithm so you don't need a single 300MB data structure.

I simplified the problem to get answers to what I needed to learn (i.e.
answers on memory management).
The original code used the MemoryStream. The buffer of the stream was
failing to extend when the buffer grew over certain size (140MB).
MemoryStream internally uses set_Capacity() which in turn creates a new
bigger (twice as big, i.e. 280MB) contiguous buffer.
I already knew that we must get rid of allocating so huge memory stream. But
I didn't understand the maths. Now I know we were comparing wrong numbers.

Thank you,
Martin.

 

[Peter Duniho]

Aha! I thought that the PerformanceCounter was giving me information about
the memory of the .NET's GC heap (which is a part of the process memory)!
Not an info about the system memory!

It's documented to provide information about available physical memory.
Physical memory is relevant only to the OS; normal processes never
"see" it (obviously they use it, but only via virtual addresses).

And I thought that the GC pre-allocates a _contiguous_ block of memory from
the system virtual memory, which it subsequently lends to managed objects. [...]

It does. It doesn't have to ask the OS for more memory for every
managed allocation. But that's not the entire story.

[...]

Firstly: for objects in the small object heap, garbage collection
_sometimes_ will defragment the heap, by compacting it. But the large
object heap is never compacted, and thus can become fragmented.

Some GC algorithms (e.g. the basic copying GC) always defragment the heap.

The .NET GC algorithm is more complex. It's main optimization is that
it's a generational collector, and older generations in the small object
heap not subject to a given collection operation are also not defragmented.

Furthermore (and this is what's important here, because you're dealing
with a large object): the large object heap is separate from the small
object heap, and it's _never_ defragmented.

But I do not know which algorithm is used by .NET. And if - as you say - the
heap memory need not be contiguous, this can influence its fragmentation
too. I believe you are right that the .NET GC algorithm need not defragment
the heap. [...]

Allocations made from the OS to increase the heap size allocate single
blocks at a time. Obviously, within each block, the available memory
addresses are contiguous. There can be fragmentation temporarily in the
small object heap, and in the large object heap, any fragmentation that
occurs will persist until the objects that are in the middle of free
space are themselves released. No data in the large object heap will be
compacted.

For all intents and purposes, you can ignore the small object heap and
its behavior. Even though it does get compacted on a regular basis,
that doesn't matter. Your data isn't being allocated there. It's being
allocated in a heap that never is compacted and so, depending on your
exact allocation patterns, can become fragmented.

Pete

 

[Martin Plechsmid]

Thank you, Peter. You've been very helpful and your explanations very
informative. Thanks again.

Martin.

Aha! I thought that the PerformanceCounter was giving me information
about the memory of the .NET's GC heap (which is a part of the process
memory)! Not an info about the system memory!

It's documented to provide information about available physical memory.
Physical memory is relevant only to the OS; normal processes never "see"
it (obviously they use it, but only via virtual addresses).

And I thought that the GC pre-allocates a _contiguous_ block of memory
from the system virtual memory, which it subsequently lends to managed
objects. [...]

It does. It doesn't have to ask the OS for more memory for every managed
allocation. But that's not the entire story.

[...]

Firstly: for objects in the small object heap, garbage collection
_sometimes_ will defragment the heap, by compacting it. But the large
object heap is never compacted, and thus can become fragmented.

Some GC algorithms (e.g. the basic copying GC) always defragment the
heap.

The .NET GC algorithm is more complex. It's main optimization is that
it's a generational collector, and older generations in the small object
heap not subject to a given collection operation are also not
defragmented.

Furthermore (and this is what's important here, because you're dealing
with a large object): the large object heap is separate from the small
object heap, and it's _never_ defragmented.

But I do not know which algorithm is used by .NET. And if - as you say -
the heap memory need not be contiguous, this can influence its
fragmentation too. I believe you are right that the .NET GC algorithm
need not defragment the heap. [...]

Allocations made from the OS to increase the heap size allocate single
blocks at a time. Obviously, within each block, the available memory
addresses are contiguous. There can be fragmentation temporarily in the
small object heap, and in the large object heap, any fragmentation that
occurs will persist until the objects that are in the middle of free space
are themselves released. No data in the large object heap will be
compacted.

For all intents and purposes, you can ignore the small object heap and its
behavior. Even though it does get compacted on a regular basis, that
doesn't matter. Your data isn't being allocated there. It's being
allocated in a heap that never is compacted and so, depending on your
exact allocation patterns, can become fragmented.

Pete

 

[Jani Järvinen [MVP]]

Hi Martin,

in addition to what others have already said, I can confirm that .NET memory
management is not a trivial topic. If you need to access large amounts of
memory, you might try to use memory mapped files. A while a go I wrote an
article about this to Developer.com, search for it or let me know and I can
share a link.

Additionally, I can share links to some good discussion about the issue you
are seeing. Browse through these, and let us know if you need further
assistance.

http://social.msdn.microsoft.com/Forums/en-US/clr/thread/b4839085-772b-4925-a0f6-2fe21096c376/

http://msdn.microsoft.com/en-us/magazine/cc163528.aspx

http://blogs.technet.com/markrussinovich/archive/2008/11/17/3155406.aspx
(from Win32 perspective)

Thank you!

--
Regards,

Jani Järvinen
C# MVP
Vantaa, Finland
E-mail: (e-mail address removed)
Business: http://www.nimacon.net/
Personal: http://www.saunalahti.fi/janij/

 

[Martin Plechsmid]

Thank you, I'll read that.

Martin.

 

[Wilson, Phil]

You're assuming that the Performance counter you're using is returning .NET
memory. You need .NET counters for .NET memory. Run Perfmon, and look at the
..NET CLR Memory counters, such as "Bytes in all Heaps".