How to compute CRC

  • Thread starter Thread starter Bob Altman
  • Start date Start date
B

Bob Altman

I am processing a very large (20+ GB) file. As part of my processing I need
to compute a CRC or hash value for the file. To improve the efficiency of
my algorithm I'm looking for a way to do this without having to read the
file twice (once for my processing and again to compute the hash).

My processing involves reading the file one 64K byte chunk at a time into a
byte array. As I'm reading the file I'd like to generate a running CRC or
hash code that I can update each time I read another bunch of bytes from the
file. Is there a way to do this without writing my own CRC32
implementation?

Thanks!

- Bob
 
Bob said:
I am processing a very large (20+ GB) file. As part of my processing
I need to compute a CRC or hash value for the file. To improve the
efficiency of my algorithm I'm looking for a way to do this without
having to read the file twice (once for my processing and again to
compute the hash).
My processing involves reading the file one 64K byte chunk at a time
into a byte array. As I'm reading the file I'd like to generate a
running CRC or hash code that I can update each time I read another
bunch of bytes from the file. Is there a way to do this without
writing my own CRC32 implementation?
Click to expand...

Yes.

While the .NET framework doesn't include a CRC32 implementation, there are
many articles/libraries available that do provide just one you're looking
for.

Here's one:

http://www.codeproject.com/csharp/crc32_dotnet.asp?df=100&forumid=4026&exp=0&select=913712

that exposes CRC32 as a class derived from
System.Security.Cryptography.HashAlgorithm. You can easily hash an
artibrarily large object in blocks as you read it in and process it.

If you don't like that one, do a few network searches - there are literally
dozens of implementations out there.

-cd
 
Thanks for the instant answer! I guess my question should have been phrased
a little more generally: Is there a way to use any of the existing hash
algorithms to compute a hash for an arbitrarily large file in blocks as I
read and process it?
 
You could implement it as a stream reader, e.g. you make a class that
you use to read the stream, and the class calculates the hash as it
reads from the stream.
 
Bob said:
Thanks for the instant answer! I guess my question should have been
phrased a little more generally: Is there a way to use any of the
existing
hash algorithms to compute a hash for an arbitrarily large file in blocks
as I read and process it?
Click to expand...

Yes. Take the CRC implementation that's a HashAlgorithm and use it as the
"Transform" for a System.Security.CryptoStream. If you don't care that it's
CRC32 in particular, the .NET framekwork already provides SHA1 and MD5 (and
other) hashes as HashTransforms.

-cd
 
Back
Top