I
Ian Taite
Hello,
I'm exploring why one of my C# .NET apps has "high" memory usage, and
whether I can reduce the memory usage.
I have an app that wakes up and processes text files into a database
periodically. What happens, is that the app reads the contents of a
text file line by line into an ArrayList. Each element of the ArrayList
is a string representing a record from the file. The ArrayList is then
processed, and the arraylist goes out of scope.
As the file is read, and the ArrayList is extended, memory usage goes
from about 6Mb up to about 200Mb in some cases, depending on the size
of the file.
What is peculiar is that after the file has been read and closed, and
the ArrayList has gone out of scope, the memory usage does not not
reduce even after waiting several minutes.
My expectation is that most of the 200Mb would be garbage collected
soon after the ArrayList goes out of scope, and released back to the
system after several minutes, but this does not happen.
Another peculiar behaviour is that when the app wakes to process a
second file, memory usage remains at about 200Mb. After the second file
is processed, memory is still 200Mb. Note that when the second file
processed, memory does not jump from 200mb to 400mb back to 200mb.
I have tried using ArrayList's Clear() method to empty the arraylist
before it goes out of scope but that does not make any difference.
To try and make the problem reproducable, I wrote a test app that
simply reads a file into an arraylist, calls ArrayList.Clear() then
allows the ArrayList to fall out of scope, and sure enough the problem
is still there. Memory usage climbs in rough proportion to the file
size and remains there until the app ends, even if the file is read
several times, and even when the array list is out of cope, i.e. not
referenced.
Explicitly calling CG.Collect() made hardly any difference, although
calling GC.GetTotalMemory(true) shortly after processing each file did
reduce the amount of memory shown by task manager for the process.
I've re-written my app not to read the whole file into memory before
processing it (not a particularly good idea in the first place) and the
maximum memory usage has changed from 200mb to 35mb.
So what is really happening here?
In C# there's no way to explicitly release resources, so I do not have
direct control over releasing memory.
Assuming that the objects in the arraylist and the arraylist itself are
no longer referenced they should be candidates for garbage collection,
and should eventually be garbage collected, at which point the memory
they used should be released back to the system.
Perhaps these objects are no longer referenced, and the GC has decided
that as there's insufficient contention for memory on the server, it
won't bother garbage collecting them, resulting in apparently high
memory usage by the application.
I don't really know how to measure what's happening, any offers would
be greatfully received. I've included some test code below in the form
of a console app; you'll have to make the code process a suitably large
text file of your own though.
Regards,
Ian.
<!-- Code Starts here -->
using System;
using System.IO;
using System.Collections;
namespace SimpleFileReader
{
class SimpleFileReader
{
public static string readLineResponse;
[STAThread]
static void Main(string[] args)
{
Console.WriteLine( "Press return to create an app object, and run
the app." );
readLineResponse = Console.ReadLine();
SimpleFileReader app = new SimpleFileReader();
app.Execute();
Console.WriteLine( "Type quit to terminate the app." );
readLineResponse = "continue";
while ( true )
{
Console.WriteLine(
"\nHit return to continue, \n" +
" or type 'gc' to garbage collect, \n" +
" or type 'gtm' to call GC.GetTotalMemory(true), \n" +
" or type 'quit' and hit return.\n\n" );
readLineResponse = Console.ReadLine();
if ( "gc" == readLineResponse ) GC.Collect();
if ( "gtm" == readLineResponse ) GC.GetTotalMemory( true );
if ( "quit" == readLineResponse ) break;
}
}
public void Execute()
{
GetRecords( @"C:\SimpleFileReader\SLEXTRACTFILE050701.DAT" );
}
public void GetRecords( string fileName )
{
int recordCounter = 0;
ArrayList al = new ArrayList();
try
{
FileStream fs = File.OpenRead( fileName );
StreamReader sr = new StreamReader( fs );
string aLine = string.Empty;
Console.WriteLine( "Press return to read the file." );
readLineResponse = Console.ReadLine();
while( ( aLine = sr.ReadLine() )!= null )
{
if ( aLine.Length > 0 )
{
al.Add( aLine );
++recordCounter;
}
}
Console.WriteLine( recordCounter.ToString() + " record(s) read from
" + fileName );
Console.WriteLine( "File read. Press return to continue." );
readLineResponse = Console.ReadLine();
sr.Close();
fs.Close();
}
catch( System.Exception ex )
{
string extractFailedMessage =
"An exception occurred whilst extracting the contents of " +
fileName;
Console.WriteLine( extractFailedMessage + "\n" + ex.ToString() );
}
Console.WriteLine( "Press return to clear the arraylist" );
readLineResponse = Console.ReadLine();
al.Clear();
Console.WriteLine( "al.Clear() called." );
Console.WriteLine( "Press return to call GC.GetTotalMemory" );
readLineResponse = Console.ReadLine();
Console.WriteLine( "GC.GetTotalMemory just returned: " +
GC.GetTotalMemory( true ).ToString() );
}
}
}
<!-- Code Ends here -->
I'm exploring why one of my C# .NET apps has "high" memory usage, and
whether I can reduce the memory usage.
I have an app that wakes up and processes text files into a database
periodically. What happens, is that the app reads the contents of a
text file line by line into an ArrayList. Each element of the ArrayList
is a string representing a record from the file. The ArrayList is then
processed, and the arraylist goes out of scope.
As the file is read, and the ArrayList is extended, memory usage goes
from about 6Mb up to about 200Mb in some cases, depending on the size
of the file.
What is peculiar is that after the file has been read and closed, and
the ArrayList has gone out of scope, the memory usage does not not
reduce even after waiting several minutes.
My expectation is that most of the 200Mb would be garbage collected
soon after the ArrayList goes out of scope, and released back to the
system after several minutes, but this does not happen.
Another peculiar behaviour is that when the app wakes to process a
second file, memory usage remains at about 200Mb. After the second file
is processed, memory is still 200Mb. Note that when the second file
processed, memory does not jump from 200mb to 400mb back to 200mb.
I have tried using ArrayList's Clear() method to empty the arraylist
before it goes out of scope but that does not make any difference.
To try and make the problem reproducable, I wrote a test app that
simply reads a file into an arraylist, calls ArrayList.Clear() then
allows the ArrayList to fall out of scope, and sure enough the problem
is still there. Memory usage climbs in rough proportion to the file
size and remains there until the app ends, even if the file is read
several times, and even when the array list is out of cope, i.e. not
referenced.
Explicitly calling CG.Collect() made hardly any difference, although
calling GC.GetTotalMemory(true) shortly after processing each file did
reduce the amount of memory shown by task manager for the process.
I've re-written my app not to read the whole file into memory before
processing it (not a particularly good idea in the first place) and the
maximum memory usage has changed from 200mb to 35mb.
So what is really happening here?
In C# there's no way to explicitly release resources, so I do not have
direct control over releasing memory.
Assuming that the objects in the arraylist and the arraylist itself are
no longer referenced they should be candidates for garbage collection,
and should eventually be garbage collected, at which point the memory
they used should be released back to the system.
Perhaps these objects are no longer referenced, and the GC has decided
that as there's insufficient contention for memory on the server, it
won't bother garbage collecting them, resulting in apparently high
memory usage by the application.
I don't really know how to measure what's happening, any offers would
be greatfully received. I've included some test code below in the form
of a console app; you'll have to make the code process a suitably large
text file of your own though.
Regards,
Ian.
<!-- Code Starts here -->
using System;
using System.IO;
using System.Collections;
namespace SimpleFileReader
{
class SimpleFileReader
{
public static string readLineResponse;
[STAThread]
static void Main(string[] args)
{
Console.WriteLine( "Press return to create an app object, and run
the app." );
readLineResponse = Console.ReadLine();
SimpleFileReader app = new SimpleFileReader();
app.Execute();
Console.WriteLine( "Type quit to terminate the app." );
readLineResponse = "continue";
while ( true )
{
Console.WriteLine(
"\nHit return to continue, \n" +
" or type 'gc' to garbage collect, \n" +
" or type 'gtm' to call GC.GetTotalMemory(true), \n" +
" or type 'quit' and hit return.\n\n" );
readLineResponse = Console.ReadLine();
if ( "gc" == readLineResponse ) GC.Collect();
if ( "gtm" == readLineResponse ) GC.GetTotalMemory( true );
if ( "quit" == readLineResponse ) break;
}
}
public void Execute()
{
GetRecords( @"C:\SimpleFileReader\SLEXTRACTFILE050701.DAT" );
}
public void GetRecords( string fileName )
{
int recordCounter = 0;
ArrayList al = new ArrayList();
try
{
FileStream fs = File.OpenRead( fileName );
StreamReader sr = new StreamReader( fs );
string aLine = string.Empty;
Console.WriteLine( "Press return to read the file." );
readLineResponse = Console.ReadLine();
while( ( aLine = sr.ReadLine() )!= null )
{
if ( aLine.Length > 0 )
{
al.Add( aLine );
++recordCounter;
}
}
Console.WriteLine( recordCounter.ToString() + " record(s) read from
" + fileName );
Console.WriteLine( "File read. Press return to continue." );
readLineResponse = Console.ReadLine();
sr.Close();
fs.Close();
}
catch( System.Exception ex )
{
string extractFailedMessage =
"An exception occurred whilst extracting the contents of " +
fileName;
Console.WriteLine( extractFailedMessage + "\n" + ex.ToString() );
}
Console.WriteLine( "Press return to clear the arraylist" );
readLineResponse = Console.ReadLine();
al.Clear();
Console.WriteLine( "al.Clear() called." );
Console.WriteLine( "Press return to call GC.GetTotalMemory" );
readLineResponse = Console.ReadLine();
Console.WriteLine( "GC.GetTotalMemory just returned: " +
GC.GetTotalMemory( true ).ToString() );
}
}
}
<!-- Code Ends here -->