OOM error with lot of available process memory

  • Thread starter Thread starter Beorne
  • Start date Start date
B

Beorne

I'm using WindowsCE 5.0 with CF 2.0

I'm trying to detect the available memory so to detect when the memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole memory.

I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).

for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless

byte[] array = new byte[1024 * i * 32];

// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;

Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";

Console.WriteLine(i + "\n" + str.ToString());
}

The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:

303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----

Why?

Thanks.
 
Because you're out of virtual memory. Your app does not have access to all
of the device memory, and if you're allocating from the stack (as opposed to
the GC heap) then it comes from process memory.

There are plenty of online resources that explain CE's memory model as well
as that of the CF.


--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded community
http://community.OpenNETCF.com
 
Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.


Because you're out of virtual memory.  Your app does not have access toall
of the device memory, and if you're allocating from the stack (as opposedto
the GC heap) then it comes from process memory.

There are plenty of online resources that explain CE's memory model as well
as that of the CF.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com


I'm using WindowsCE 5.0 with CF 2.0
Click to expand...
I'm trying to detect the available memory so to detect when the memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole memory.
Click to expand...
I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).
Click to expand...
           for (int i = 1; i < 100000; i ++)
           {
               GC.Collect(); // probably useless
Click to expand...
               byte[] array = new byte[1024 * i * 32];
Click to expand...
               // probably useless
               for (int j = 0; j < array.Length; j++)
                   array[j] = (byte)0;
Click to expand...
               Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
               str =
                   "GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
                   "ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
                   "ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
                   "ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
                   "ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
                   "ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";
Click to expand...
               Console.WriteLine(i + "\n" + str.ToString());
           }
Click to expand...
The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:
Click to expand...
303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----

Thanks.
Click to expand...
Click to expand...
 
OK, so what's the question then?

if your last output was 303, then the next attempt to allocate will be

1024 * 304 * 32 == 9,961,472 bytes.

You have 9.7MB free and last I checked 9.9 (your attempt) is greater than
9.7 (what's available), so not surprisingly you get an error.

Your code doesn't output the failure values, plus what you're doing is going
to require contiguous memory, and you certainly can't assume that the number
reported as available is all contiguous.


--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded community
http://community.OpenNETCF.com


Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.


Because you're out of virtual memory. Your app does not have access to all
of the device memory, and if you're allocating from the stack (as opposed
to
the GC heap) then it comes from process memory.

There are plenty of online resources that explain CE's memory model as
well
as that of the CF.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com


I'm using WindowsCE 5.0 with CF 2.0
Click to expand...
I'm trying to detect the available memory so to detect when the memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole memory.
Click to expand...
I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).
Click to expand...
for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless
Click to expand...
byte[] array = new byte[1024 * i * 32];
Click to expand...
// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;
Click to expand...
Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";
Click to expand...
Console.WriteLine(i + "\n" + str.ToString());
}
Click to expand...
The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:
Click to expand...
303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----

Thanks.
Click to expand...
Click to expand...
 
The question is:
why the gc does not collect the previous out of scope array before
trying to commit the remaining available virtual memory?

OK, so what's the question then?

if your last output was 303, then the next attempt to allocate will be

1024 * 304 * 32 == 9,961,472 bytes.

You have 9.7MB free and last I checked 9.9 (your attempt) is greater than
9.7 (what's available), so not surprisingly you get an error.

Your code doesn't output the failure values, plus what you're doing is going
to require contiguous memory, and you certainly can't assume that the number
reported as available is all contiguous.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com


Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.

Because you're out of virtual memory. Your app does not have access to all
of the device memory, and if you're allocating from the stack (as opposed
to
the GC heap) then it comes from process memory.
Click to expand...
There are plenty of online resources that explain CE's memory model as
well
as that of the CF.

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com
Click to expand...
news:[email protected]...
Click to expand...
I'm using WindowsCE 5.0 with CF 2.0
I'm trying to detect the available memory so to detect when the memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole memory.
I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).
for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless
byte[] array = new byte[1024 * i * 32];
// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;
Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";
Console.WriteLine(i + "\n" + str.ToString());
}
The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:
303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----
Why?
Thanks.
Click to expand...
Click to expand...
Click to expand...
 
How much VM is available before you make any of these allocations?
Have you tried to catch the OOM, call Collect in the handler and
re-allocate?
Why does this even matter? Unless you have an app that does nothing but
generate these gigantic arrays, I don't see the entire point of this
exercise. You must be trying to solve some real problem, right? What
problem is that? If you need a 10MB contiguous array for something, there
are probably better ways to allocate.


--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded community
http://community.OpenNETCF.com


Beorne said:
The question is:
why the gc does not collect the previous out of scope array before
trying to commit the remaining available virtual memory?

OK, so what's the question then?

if your last output was 303, then the next attempt to allocate will be

1024 * 304 * 32 == 9,961,472 bytes.

You have 9.7MB free and last I checked 9.9 (your attempt) is greater than
9.7 (what's available), so not surprisingly you get an error.

Your code doesn't output the failure values, plus what you're doing is
going
to require contiguous memory, and you certainly can't assume that the
number
reported as available is all contiguous.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com


Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.

Because you're out of virtual memory. Your app does not have access to
all
of the device memory, and if you're allocating from the stack (as
opposed
to
the GC heap) then it comes from process memory.
Click to expand...
There are plenty of online resources that explain CE's memory model as
well
as that of the CF.

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com
Click to expand...
I'm using WindowsCE 5.0 with CF 2.0
Click to expand...
I'm trying to detect the available memory so to detect when the
memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole
memory.
Click to expand...
I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).
Click to expand...
for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless
Click to expand...
byte[] array = new byte[1024 * i * 32];
Click to expand...
// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;
Click to expand...
Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";
Click to expand...
Console.WriteLine(i + "\n" + str.ToString());
}
Click to expand...
The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:
Click to expand...
303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----

Thanks.
Click to expand...
Click to expand...
Click to expand...
 
You are right, allocating every cycle a byte[1024 * 32] I don't
claim contiguous memory and I can reach a wonderful
ms.dwAvailVirtual = 0
before going oom

Thank you!


How much VM is available before you make any of these allocations?
Have you tried to catch the OOM, call Collect in the handler and
re-allocate?
Why does this even matter?  Unless you have an app that does nothing but
generate these gigantic arrays, I don't see the entire point of this
exercise.  You must be trying to solve some real problem, right?  What
problem is that?  If you need a 10MB contiguous array for something, there
are probably better ways to allocate.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com


The question is:
why the gc does not collect the previous out of scope array before
trying to commit the remaining available virtual memory?
Click to expand...
OK, so what's the question then?
if your last output was 303, then the next attempt to allocate will be
1024 * 304 * 32 == 9,961,472 bytes.
You have 9.7MB free and last I checked 9.9 (your attempt) is greater than
9.7 (what's available), so not surprisingly you get an error.
Your code doesn't output the failure values, plus what you're doing is
going
to require contiguous memory, and you certainly can't assume that the
number
reported as available is all contiguous.
--
Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com
Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.
On 8 Apr, 14:37, "Chris Tacke, MVP" <ctacke.at.opennetcf.dot.com>
wrote:
Because you're out of virtual memory. Your app does not have access to
all
of the device memory, and if you're allocating from the stack (as
opposed
to
the GC heap) then it comes from process memory.
There are plenty of online resources that explain CE's memory model as
well
as that of the CF.
--
Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com

I'm using WindowsCE 5.0 with CF 2.0
I'm trying to detect the available memory so to detect when the
memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole
memory.
I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).
for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless
byte[] array = new byte[1024 * i * 32];
// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;
Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";
Console.WriteLine(i + "\n" + str.ToString());
}
The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:
303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----
Why?
Thanks.
Click to expand...
Click to expand...
Click to expand...
 
Back
Top