Process File in Chunks - StreamReader

  • Thread starter Thread starter das
  • Start date Start date
D

das

Hi All,
I want to process a file in chunks, meaning I have a big file of
50K rows. I want to process 5K rows at a time.

I am using StreamReader object and looping through each row and process
it, and as I process it I create an XML file and send it to DB to be
processed.

while(oSR.Peek() > -1)
{
// 1. Create Xml from these lines
}
// 2. Call DB Functions.

How do I read only certain number of lines at a time, complete my steps
1 and 2 and come back and read another 5K lines and so on....

any ideas gurus?
thanks
 
Hi All,
I want to process a file in chunks, meaning I have a big file of
50K rows. I want to process 5K rows at a time.

I am using StreamReader object and looping through each row and process
it, and as I process it I create an XML file and send it to DB to be
processed.

while(oSR.Peek() > -1)
{
// 1. Create Xml from these lines
}
// 2. Call DB Functions.

How do I read only certain number of lines at a time, complete my steps
1 and 2 and come back and read another 5K lines and so on....

any ideas gurus?
thanks
Click to expand...

This will do what you want...

using System;
using System.Collections.Generic;
using System.Text;
using System.Data;

namespace TestConsole
{
class Program
{
static void Main(string[] args)
{
// simulating a file read here
int i = 0;
int j = 0;
while (i < 50000)
{
if (j < 5000)
{
// do your xml stuff
j++;
i++;
}
else
{
Console.WriteLine("i = {0}, j = {1}", i, j);
j = 0;
Console.WriteLine("\tDoing DB stuff");
}
}
Console.WriteLine("Finally i = {0}, j = {1}", i, j);
Console.WriteLine("\tDoing DB stuff");
Console.Write("Press any key to close:");
Console.ReadLine();
}
}
}

Otis Mukinfus
http://www.arltex.com
http://www.tomchilders.com
 
Back
Top