Out of memory error

  • Thread starter Thread starter Schoo
  • Start date Start date
S

Schoo

I have a simple ASPX page that uses 'response.write' to take data from a
dataset, scroll through it and write it back to the page as HTML. This
works just great, until you get too many records in the dataset (it fails
when there is about 575 records in the dataset). The application event log
on the server (Windows 2000) says: "aspnet_wp.exe (PID:...) was recycled
because memory consumption exceeded the 153 MB (60 percent of available
RAM). When this goes into production, I expect the database to contain
hundreds of thousands of records (SQL 2000).

I have developed other db apps so I don't think it is the dataset, in fact I
believe I have established that the problem is that the server will not
allow my app to write all that information (strings) to the html page. Is
there a way to commit all the strings as it writes them to the page? Is
there a setting to allow the memory to expand? Is there some other
technique I should be using to do this?

Scott
 
Hi Scott,

Thanks for posting in the community!
From your description,you used Response.Write to render a certain page's
output content from a certain
dataset which contains hundreds of records. Also you used the string to
manipulate the output content, however,
when running, you encoutnered out of memory exception. And from the
eventlog , you found that
it did caused by the memory useage exceed 60% of the avaliable physical
memory and the aspnet worker process
recycled. So you wonder :
1. how to allow him still wirte out all the strings generated from the
dataset to the output stream
2. change the limiation of the memory useage of the aspnet worker process
If there is anything I misunderstood, please feel free to let me know.

As for this problem, here are my suggstions:
1. From the description proivded by your eventlog. It occured when 153 MB
memory is used and exceed the 60% limitation. Does your machine have 256MB
physical memory installed? If so, we can confirm that the problem(out of
menory exception and workerprocess recycle) is really caused by the memory
useage. And this memory usage limitation can be configured in the
machine.config file within the <processModel> element, by default, its
value is 60%, you can adjust it to larger percent value. You may get the
detailed info on this via the following reference in MSDN:
#<processModel> Element
http://msdn.microsoft.com/library/en-us/cpgenref/html/gngrfProcessmodelSecti
on.asp?frame=true

2. Though we can change the memory usage limitation of a aspnet
workerprocess in the above element, I still think this problem is caused by
the certain performance issue with in the page which render out the content
from dataset. As you mentioned that the DataSet has great amount of
recoreds and you generate the whole content using the string class, yes?
Then, do you use many manipulations such as appending string or resize it
by concat different strings, since the string class's instance is
unresizeable, such operations will cause a new string instance be created
and replace the original one, this will cause poor performance and large
memory useage. I suspect whether this is a potential cause of this issue?
And there is a class named "StringBuilder" in the System.Text namespace,
which is helped to provide string manipulate operations and won't
reallocate new instance when size changed. You may try using StringBuilder
class to replace the string to improve your performance. Here are some tech
articles on using stringbuilder rather than string to improve performance:

#Improving String Handling Performance in .NET Framework Applications
http://msdn.microsoft.com/library/en-us/dndotnet/html/vbnstrcatn.asp?frame=t
rue

#Manipulate Strings Faster in VB.
http://msdn.microsoft.com/library/en-us/dnvbpj01/html/gs0107.asp?frame=true

#Using the StringBuilder Class
http://msdn.microsoft.com/library/en-us/cpguide/html/cpconusingstringbuilder
class.asp?frame=true

Please check out the above suggestions and references. If you have any
further questions, please feel free to post here.


Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx
 
Scott,

even though you can modify the <processModel> setting to give your process
more
memory before it will be recycled, I think you should consider using a
DataReader
instead of a DataSet to read from your database.

From what I understand you only need sequential read access to a large
number of records.
In this case a DataReader is probably a more efficient way to solve your
problem than
reading all the records into a DataSet.

Check out the documentation on DataReaders for more details.

- Dennis
 
Dennis,

Thank you for your idea. I totally agree that a datareader would be most
appropriate here. However, I converted the procedure to use a datareader
instead of dataset and I get the same error. I think that the problem here
is that the system can't handle all the writing to the html page. Please
see the other string for more information.

Scott
 
Steven,

Thank you for your response. After pursuing your ideas for almost 3 hours
this morning here is what I have come up with:

Memory on the server is 261,648KB. However, I am not clear on which memory
the compiled code is using to run its logic. At this point I am assuming it
is the server that is running short of memory and not the client because of
the message on the server log.

In response to your 2nd point, I should have mentioned that I have been
using a stringbuilder and not a string all along.

I spent most of the time this morning testing and seeing if I can narrow
down where the problems is. I must also add that my code is not very
complicated and should not be difficult to duplicate a similar situation on
your system. Maybe if you knew more information it would help in your
diagnosing the problem:

I have a stored proc (SQL2000) returning 376 records of which I am using 6
columns. All of the columns return varchars (except one that is an
integer). I am using a datareader through the MS Application Data Block
which returns the results of the stored proc. I then scroll through the
records and form the contents of a grid using <table>, <tr>, and <td> tags.
Normally I would use a dataset and bind it to a grid control, but in this
case the specification asks for data in different cells based on the values
of other cells, which means I can't just bind the data to a grid, and also
they want it to appear as an Excel spreadsheet in the browser, so I have a
line in the Page_Load procedure that reads: "Response.ContentType =
"application/vnd.ms-excel". I DID try to pull out that line to see if it
would run but that did not work either.

Placing break-points in various places, I learned that the code apparently
does not get to the point where it returns the data to the page ("Return
strResponse.ToString"). [Yes, I converted out the "response.write
(strResponse.ToString)" to create a function instead of a procedure, but it
didn't seem to make a difference.] So, when there is 300 records that need
to be processed, it runs fine, but if there are 376 records it never gets to
the end of the 'While' loop.

This is very strange! This isn't alot of records... when this goes into
production I expect hundreds of thousands of records to scroll through.
Even if I increase memory allocation on the server and that works, it would
only be a temporary solution. Is there another technique that would present
this as an excel spreadsheet better? I have some experience creating Excel
spreadsheets programatically using VB but it's been years since I had to do
that and the method I am trying to copy here was originally ASP logic that
worked with Access which I converted to a datareader and ASP.NET with a SQL
Server 2000 datasource. This shouldn't be this difficult... the answer
should be obvious, so what am I doing wrong?

Scott
 
Hi Scott,

Thanks for your response. As for how to export datas as excel style in
ASP.NET web pages. Here is tech article on exporting data into Excel sheet
in ASP.NET web pages. It has discussed serveral different means:

#Export to Excel
http://www.aspnetpro.com/NewsletterArticle/2003/09/asp200309so_l/asp200309so
_l.asp

As for the code's performance and the memory usage, I also think there is
still some means to improve it. But I've not tested it and not sure whether
256MB memory is a bit tight for testing. Any way, since you 're still
testing, I think you can do some further monitor on the web app's
performance and memory useage. Here are some tech references on Monitoring
the ASP.NET web app's performance and memory usage:

#ASP.NET Performance Monitoring, and When to Alert Administrators
http://msdn.microsoft.com/library/en-us/dnaspp/html/monitor_perf.asp?frame=t
rue

#Production Debugging for .NET Framework Applications
http://msdn.microsoft.com/library/en-us/dnbda/html/DBGch02.asp?frame=true

#Performance Counters for ASP.NET
http://msdn.microsoft.com/library/en-us/cpguide/html/cpconperformancecounter
sforaspnet.asp?frame=true

#ASP.NET Optimization
http://msdn.microsoft.com/library/en-us/cpguide/html/cpconaspoptimization.as
p?frame=true

Also, some has discussed on the optimzation. Hope they're helpful.


Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx
 
Steven,

The aspnetpro article was tricky because their examples do not outline how
to pull actual (or demo) data into their example but I used some info from
other sites to pull that together.

I also had problems with setting up the data columns. msdn outlines 2
methods to do this but neither worked for me the way MS wrote them:

dt.columns.add("lname", type.gettype("system.string")

I keep getting the error message: "data type arguement cannot be null..."
So I dropped the 2nd arguement and it worked fine. Not the proper way to do
it, but it got me by. Please feel free to suggest a reason why this
statement doesn't work.

In the end, I got it. I used the 2nd method from the aspnetpro article and
I will study the other links that you provided as I am weak in those areas.
Thank you for your help. :)

Scott
 
Hi Scott,

Thanks for your followup. I'm glad that my suggestions has helped you. As
for the Exception you encountered when adding column to the DataTable, I
think it is cause by the "system.string" you passed to the Type.GetType
method,
in dotnet framework, the class name is case sensitive no matter you using
C# or VB.NET.
I've tested in C# via the below code and it works ok:

tb.Columns.Add("type1",typeof(string));
tb.Columns.Add("type2",Type.GetType("System.String"));

Please also have a try to see whether it works for you. Also, if you have
any further questions, please feel free to let me know.


Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx
 
Hi Scott,

Have you got any progress on this issue? Here is some additional
suggestions for troubleshooting:
I suggest that the you try reducing the output as an experiment. Whatever
you are putting into this string that you're building (table with rows &
cells probably), change it to output one character for each row of data.
Does the problem occur? If the problem still occurs, then focus on the
database functions. If the problem stops occurring, then experiment with
outputting increasing amounts of text to see where it breaks. Hope this
helps.


Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx
 
Steven,

The aspnetpro article was tricky because their examples do not outline how
to pull actual (or demo) data into their example but I used some info from
other sites to pull that together.

I also had problems with setting up the data columns. msdn outlines 2
methods to do this but neither worked for me the way MS wrote them:

dt.columns.add("lname", type.gettype("system.string")

I keep getting the error message: "data type arguement cannot be null..."
So I dropped the 2nd arguement and it worked fine. Not the proper way to do
it, but it got me by. Please feel free to suggest a reason why this
statement doesn't work.

In the end, I got it. I used the 2nd method from the aspnetpro article and
I will study the other links that you provided as I am weak in those areas.
Thank you for your help. :)

Scott
 
Back
Top