Large Datasets

  • Thread starter Thread starter Bernie Hunt
  • Start date Start date
B

Bernie Hunt

Has anyone seen any documentation or white papers with suggestions on how
to handle large datasets. My task is to read in 10K records and then step
through each one processing it's data. The processing involved fetching 3
other records that match the current, manipulate the data and then write
out 5 records to a different database.

DataReader is out of the question, because this whole precessing time could
take 10 or 20 minutes, so it looks like a DataSet. My concern is what
happens if a DataSet exceeds memory. Is it cached on disk in the swapfile
or does .Net handle it's temp storage. I'd love to find some kind of
guideline on how large can a DataSet get without causing calaterail damage
in the environment.

Thanks for any guidance.

Bernie
 
Hi Bernie,

Is server processing an option for you? (stored procedure or something)
What databaes are we talking?
 
Bernie,

A dataset can be as large as the program leaves enough memory for it. Where
the program can be AFAIK not larger than 800Mb.

I hope this helps

Cor
 
Bernie Hunt said:
Has anyone seen any documentation or white papers with suggestions on how
to handle large datasets. My task is to read in 10K records and then step
through each one processing it's data. The processing involved fetching 3
other records that match the current, manipulate the data and then write
out 5 records to a different database.

DataReader is out of the question, because this whole precessing time
could
take 10 or 20 minutes, so it looks like a DataSet. My concern is what
happens if a DataSet exceeds memory. Is it cached on disk in the swapfile
or does .Net handle it's temp storage. I'd love to find some kind of
guideline on how large can a DataSet get without causing calaterail damage
in the environment.
Click to expand...

10,000 rows in a DataSet really isn't all that much. Just try it.

David
 
Hi Cor,

Cor Ligthert said:
Bernie,

A dataset can be as large as the program leaves enough memory for it.
Where the program can be AFAIK not larger than 800Mb.
Click to expand...

Where did you get this number from? Voodoo?
 
I could discuss here, but there are 64 bit CPUs out there and there is .net
2 that manages memory better.
Ah, and if I recall correctly, there is a switch that lets you use more than
2GB memory for user appsin 32bit windows, too.
And 800Mb is not accurate - it is just an assumption.
 
I could discuss here, but there are 64 bit CPUs out there and there is .net
2 that manages memory better.
Ah, and if I recall correctly, there is a switch that lets you use more
than
2GB memory for user appsin 32bit windows, too.
And 800Mb is not accurate - it is just an assumption.
Click to expand...


Did I write something else?

Although the by you mentioned switch (the name is 3Gb switch) seems not
affect this behaviour it stays aprox 800Mb per process.

Cor
 
Cor Ligthert said:
Did I write something else?
Click to expand...

" Where the program can be AFAIK not larger than 800Mb."

Note: 64bit CPUs can address terabytes of memory.
 
" Where the program can be AFAIK not larger than 800Mb."

Note: 64bit CPUs can address terabytes of memory.
Click to expand...
Yes I wrote As Far As I Know.

Can you than give it for a 64bit CPU. Because this is not a memory behaviour
but an OS behaviour and than even as it is written with the layer Net 1.x.

However as you can enlighten me with some more precise figurs for a 64bit XP
OS with Net 2.0 and/or Vista than I will be delighted.

Cor
 
Cor:

I did some extensive testing on this a couple of years back with the 1.1
..Net Framework and console/web services. The /3GB switch *might* affect
the amount of memory a .Net application can use. It really depends on
the flavor of Operating System.


For example, a test app on 32-bit Windows XP or Windows 2003 Standard
Server (using /3GB) will only allocate an additional 900MB - 1200 MB of
memory. But on 32-bit Windows 2003 Enterprise Server, using the /3GB
switch will provide an application with the additional memory. Note, the
/3GB switch is a double edged sword as it will limit any OS processes max
memory size to use 1GB.


Some useful info from the original thread (watch word wrap) -

http://groups.google.com/group/microsoft.public.dotnet.framework.aspnet.w
ebservices/tree/browse_frm/thread/cf6567aa5b6ee5d8/265abc31aa209df0?rnum=
1&hl=en&q=%2F3gb+clausius&_done=%2Fgroup%
2Fmicrosoft.public.dotnet.framework.aspnet.webservices%2Fbrowse_frm%
2Fthread%2Fcf6567aa5b6ee5d8%2F265abc31aa209df0%3Ftvc%3D1%26q%3D%
2F3gb+clausius%26hl%3Den%26#doc_265abc31aa209df0


Jeff Clausius
SourceGear
 
Jeff,

Thanks,

Cor

Jeff Clausius said:
Cor:

I did some extensive testing on this a couple of years back with the 1.1
.Net Framework and console/web services. The /3GB switch *might* affect
the amount of memory a .Net application can use. It really depends on
the flavor of Operating System.


For example, a test app on 32-bit Windows XP or Windows 2003 Standard
Server (using /3GB) will only allocate an additional 900MB - 1200 MB of
memory. But on 32-bit Windows 2003 Enterprise Server, using the /3GB
switch will provide an application with the additional memory. Note, the
/3GB switch is a double edged sword as it will limit any OS processes max
memory size to use 1GB.


Some useful info from the original thread (watch word wrap) -

http://groups.google.com/group/microsoft.public.dotnet.framework.aspnet.w
ebservices/tree/browse_frm/thread/cf6567aa5b6ee5d8/265abc31aa209df0?rnum=
1&hl=en&q=%2F3gb+clausius&_done=%2Fgroup%
2Fmicrosoft.public.dotnet.framework.aspnet.webservices%2Fbrowse_frm%
2Fthread%2Fcf6567aa5b6ee5d8%2F265abc31aa209df0%3Ftvc%3D1%26q%3D%
2F3gb+clausius%26hl%3Den%26#doc_265abc31aa209df0


Jeff Clausius
SourceGear
Click to expand...
 
No, There are two applications where this applies. First the originating
database is an ODBC connection to a Faircom database. The vast majority of
the customers using this will be running XP Pro with no server.

The second application is for data conversion where the originating
database could be just about anything, Pervasive, Advantage and dBase are
the most common.

Bernie
 
Back
Top