Streaming Techniques

  • Thread starter Thread starter jp2msft
  • Start date Start date
J

jp2msft

What are the "Pros and Cons" with performing a stream operation in chunks as
opposed to performing that same operation on the full file?

"In Chunks" Example:
// *************************
void ReadFileInChunks(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
int len;
byte[] buffer = new byte[1024];
do {
len = fs.Read(buffer, 0, 1024);
Console.WriteLine("Read 1024 bytes of data.");
} while (0 < len);
fs.Close();
}
}

"Full File" Example:
// *************************
void ReadFileAtOnce(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
byte[] buffer = new byte[fs.Length];
if (fs.Read(buffer, 0, buffer.Length) == buffer.Length) {
Console.WriteLine("Finished!");
}
fs.Close();
}
}

I want to use a technique that will give me the greatest performance while
ensuring the method does not fail.

I'm guessing the ReadFileAtOnce method works fine as long as there is enough
RAM to read the entire file. (i.e. Passing ReadFileAtOnce a 6GB ZIP file
backup of a DVD would be bad.)

Any other thoughts on the subject?
 
Well, first off, your examples are wrong. You are not guaranteed to get
the number of bytes back that you request, and should account for that in
your code.

That being said, if you have a very large file, it would probably be a
little prohibitive to load all of the data into memory. I'm being ambiguous
with terms like "very large file" and "prohibitive" because it depends on
the specs of your machine, the code, etc, etc. You will have to test to
determine which is best.

Reading the entire file if the file is 6GB probably would be fine on a
64 bit machine, but on a 32 bit machine, you are going to run into definite
issues.
 
So, if I were to read or write a little bit at a time, what is a good amout
to use? 1024 (1KB)?

Nicholas Paldino said:
Well, first off, your examples are wrong. You are not guaranteed to get
the number of bytes back that you request, and should account for that in
your code.

That being said, if you have a very large file, it would probably be a
little prohibitive to load all of the data into memory. I'm being ambiguous
with terms like "very large file" and "prohibitive" because it depends on
the specs of your machine, the code, etc, etc. You will have to test to
determine which is best.

Reading the entire file if the file is 6GB probably would be fine on a
64 bit machine, but on a 32 bit machine, you are going to run into definite
issues.

--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

jp2msft said:
What are the "Pros and Cons" with performing a stream operation in chunks
as
opposed to performing that same operation on the full file?

"In Chunks" Example:
// *************************
void ReadFileInChunks(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
int len;
byte[] buffer = new byte[1024];
do {
len = fs.Read(buffer, 0, 1024);
Console.WriteLine("Read 1024 bytes of data.");
} while (0 < len);
fs.Close();
}
}

"Full File" Example:
// *************************
void ReadFileAtOnce(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
byte[] buffer = new byte[fs.Length];
if (fs.Read(buffer, 0, buffer.Length) == buffer.Length) {
Console.WriteLine("Finished!");
}
fs.Close();
}
}

I want to use a technique that will give me the greatest performance while
ensuring the method does not fail.

I'm guessing the ReadFileAtOnce method works fine as long as there is
enough
RAM to read the entire file. (i.e. Passing ReadFileAtOnce a 6GB ZIP file
backup of a DVD would be bad.)

Any other thoughts on the subject?
Click to expand...
Click to expand...
 
That depends completely on how much you need to process at a time, the
hardware of the machine you are runing on, and how much processing you have
to perform on each chunk.

You have to tweak and find out for yourself.

If your buffer is too small, you are going to have too many allocations
taking place, and that can affect performance. If the amount of memory
allocated is too large then you might have difficulties allocating the
space.

--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

jp2msft said:
So, if I were to read or write a little bit at a time, what is a good
amout
to use? 1024 (1KB)?

Nicholas Paldino said:
Well, first off, your examples are wrong. You are not guaranteed to
get
the number of bytes back that you request, and should account for that in
your code.

That being said, if you have a very large file, it would probably be
a
little prohibitive to load all of the data into memory. I'm being
ambiguous
with terms like "very large file" and "prohibitive" because it depends on
the specs of your machine, the code, etc, etc. You will have to test to
determine which is best.

Reading the entire file if the file is 6GB probably would be fine on
a
64 bit machine, but on a 32 bit machine, you are going to run into
definite
issues.

--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

jp2msft said:
What are the "Pros and Cons" with performing a stream operation in
chunks
as
opposed to performing that same operation on the full file?

"In Chunks" Example:
// *************************
void ReadFileInChunks(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
int len;
byte[] buffer = new byte[1024];
do {
len = fs.Read(buffer, 0, 1024);
Console.WriteLine("Read 1024 bytes of data.");
} while (0 < len);
fs.Close();
}
}

"Full File" Example:
// *************************
void ReadFileAtOnce(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
byte[] buffer = new byte[fs.Length];
if (fs.Read(buffer, 0, buffer.Length) == buffer.Length) {
Console.WriteLine("Finished!");
}
fs.Close();
}
}

I want to use a technique that will give me the greatest performance
while
ensuring the method does not fail.

I'm guessing the ReadFileAtOnce method works fine as long as there is
enough
RAM to read the entire file. (i.e. Passing ReadFileAtOnce a 6GB ZIP
file
backup of a DVD would be bad.)

Any other thoughts on the subject?
Click to expand...
Click to expand...
Click to expand...
 
jp2msft said:
So, if I were to read or write a little bit at a time, what is a good amout
to use? 1024 (1KB)?
Click to expand...

It used to be.

Today I would go for 10KB or 100KB.

Arne
 
Arne Vajh?j said:
It used to be.

Today I would go for 10KB or 100KB.
Click to expand...

There's a significant difference between these two, however - 100KB
ends up on the large object heap, which is never compacted (IIRC,
anyway). 10K ends up on the regular heap.

Personally I usually go for 8K or 16K; I'll try 32K or 64K for
benchmarking if appropriate; I wouldn't go above 64K without really
good evidence though.
 
Back
Top