I have a program that builds a flat file (~40MB) - lot of string
concatenations (~50 per line) and loops (~500 per record, ~500 records
per file) involved.
At first I was using strings - stringA + string B etc.... and ran into
out of memory issue
So, I switched to StringBuilder and StringBuilder.Append and still been
running into out of memory issues.
I would appreciate any ideas on how to troubleshoot and resolve the
issue
Hard to say. Given the metrics you've stated, it seems odd that you're
running into any out-of-memory problems. 40MB isn't a very large file,
and while concatenations are _slower_ as compared to using StringBuilder,
at the size of data you're talking about they shouldn't cause enough of an
increase in memory usage to cause an issue.
One potential thing to watch out for: very large objects wind up in the
"large object heap", which is never compacted and so carries a lot greater
risk of fragmentation problems. Objects larger than about 80K (85K? I
forget) will wind up in that heap.
If your strings are really getting that large (and if they are, yuck!),
then you should try to reuse a single StringBuilder instance rather than
creating a new one for each operation. That way, eventually it will get
large enough to sustain whatever operations you're going to do and your
program can stop allocating space from the LOH. If you keep creating new
StringBuilder instances for each concatenation operation, then you'll get
a bunch of different sized blocks allocated from the LOH, and depending on
the usage pattern could wind up fragmenting the LOH enough to prevent new
allocations from succeeding.
Other than that, without a concise-but-complete code example that reliably
demonstrates the problem, it's hard to say what would be the best way to
investigate the problem, never mind fix it. There are some third-party
memory profiling tools available, which might be useful to you for trying
to figure out what the root cause of your OOM issue is. That is, are you
fragmenting a heap? Running out of virtual address space? Running out of
disk space? What objects are causing the issue? Etc. But I don't have a
specific recommendation for one; you should use Google and check out
what's available.
Pete
