Handling very huge data

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

Hi All

Wish all of u Happy New Year

I am developing a application where in I retrieve data from Database which
can go up millions records, do some processing on that and update the same in
single transaction,
now with limited RAM say 500MB how should I handle this data (I know one
solution might be paging) so that my ASP.NET process does not fall short of
memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
recycled because memory consumption exceeded the 305 MB (60 percent of
available RAM). "

Regards
Makarand
 
Makarand,

It sounds to me like you have to real options here.

1) Chunk thru the data. I would choose this if there is some specific
business logic that you cannot perform directly on the database layer.

2) Process the Data in the Database. I would choose this if you can
perform the logic on the database layer. If you can go this route it
will probably be the most efficient.

HTH,
-eric
 
Sahil
Thanx for prompt reply,
Thing is I get the data from database server, add feed this data to MS
RuleEngine for applying Business Rules and after that I take same data and
update it in database.
So I don't c any scope of doing all these things in database.
Hope this time I am more clear on my problem

Makarand
 
Miha
Thanx for prompt reply,
Thing is I get the data from database server, add feed this data to MS
RuleEngine for applying Business Rules and after that I take same data and
update it in database.
So I don't c any scope of doing all these things in database.
Hope this time I am more clear on my problem

Makarand


Miha Markic said:
Did you consider using a stored procedure for processing the data?

--
Miha Markic [MVP C#] - RightHand .NET consulting & development
SLODUG - Slovene Developer Users Group
www.rthand.com

Makarand Keer said:
Hi All

Wish all of u Happy New Year

I am developing a application where in I retrieve data from Database which
can go up millions records, do some processing on that and update the same
in
single transaction,
now with limited RAM say 500MB how should I handle this data (I know one
solution might be paging) so that my ASP.NET process does not fall short
of
memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
recycled because memory consumption exceeded the 305 MB (60 percent of
available RAM). "

Regards
Makarand
Click to expand...
Click to expand...
 
Hi,

try using API server side cursors that would fetch you data in batches, so
that your app. will not block for that period, which greatly improves user
experience.

regards
Joyjit
 
Back
Top