R
Rob Corwin
Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:
- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.
- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.
I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.
I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.
Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:
- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.
- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.
I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.
I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.
Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob