How can I zip a 8 Gb file using c# ?

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

I need to zip up an 8Gb file using C# - do you know of any freeware libraries
I can use to accomplish this? I guess it needs to be Zip64 to support such
file sizes.

I tried something called sharpziplib but it doesn't support >2Gb files -
they are working on it however but I cannot wait...
 
Hello, MrNobody!

M> I need to zip up an 8Gb file using C# - do you know of any freeware
M> libraries
M> I can use to accomplish this? I guess it needs to be Zip64 to support
M> such
M> file sizes.

M> I tried something called sharpziplib but it doesn't support >2Gb
M> files -
M> they are working on it however but I cannot wait...

Did you try GZipStream?
( http://msdn2.microsoft.com/en-us/library/system.io.compression.gzipstream.aspx )
You must be using .NET 2.0 to work with it.

--
Regards, Vadym Stetsyak
www: http://vadmyst.blogspot.com
 
Vadym Stetsyak said:
Hello, MrNobody!

M> I need to zip up an 8Gb file using C# - do you know of any freeware
M> libraries
M> I can use to accomplish this? I guess it needs to be Zip64 to support
M> such
M> file sizes.

M> I tried something called sharpziplib but it doesn't support >2Gb
M> files -
M> they are working on it however but I cannot wait...

Did you try GZipStream?
( http://msdn2.microsoft.com/en-us/library/system.io.compression.gzipstream.aspx )
You must be using .NET 2.0 to work with it.
Click to expand...

Looks like GZip wouldn't work either, from the 'Remarks' :

"This class cannot be used to compress files larger than 4 GB."
 
MrNobody said:
Looks like GZip wouldn't work either, from the 'Remarks' :

"This class cannot be used to compress files larger than 4 GB."
Click to expand...

4GB is the maximum addressible size for a 32-bit pointer. Also, GZIP is not
the same as ZIP, so if you need PKZIP compatibility, then you need to get
another library.
 
|
|
| "Vadym Stetsyak" wrote:
|
| > Hello, MrNobody!
| >
| > M> I need to zip up an 8Gb file using C# - do you know of any freeware
| > M> libraries
| > M> I can use to accomplish this? I guess it needs to be Zip64 to support
| > M> such
| > M> file sizes.
| >
| > M> I tried something called sharpziplib but it doesn't support >2Gb
| > M> files -
| > M> they are working on it however but I cannot wait...
| >
| > Did you try GZipStream?
| > (
http://msdn2.microsoft.com/en-us/library/system.io.compression.gzipstream.aspx )
| > You must be using .NET 2.0 to work with it.
| >
| > --
| > Regards, Vadym Stetsyak
| > www: http://vadmyst.blogspot
|
| Looks like GZip wouldn't work either, from the 'Remarks' :
|
| "This class cannot be used to compress files larger than 4 GB."

Which is wrong too, actually the limit is 2GB, that's the max.size of an
object in .NET and the GZIP library needs the complete file to be read in a
single byte array (which obviously is limitted to 2GB).

Willy.
 
OK guys, found someone who made a modification to the sharpziplib out there
to support Zip64 and it works. Compressed my 7.6 Gb file and then used Winzip
to uncompress it. Works good !

thanks for your time trying to help me :)
 
Would you like to share the name/web address of "someone". Might be useful
to know.

Steve
 
Willy Denoyette said:
|
|
| "Vadym Stetsyak" wrote:
|
| > Hello, MrNobody!
| >
| > M> I need to zip up an 8Gb file using C# - do you know of any freeware
| > M> libraries
| > M> I can use to accomplish this? I guess it needs to be Zip64 to support
| > M> such
| > M> file sizes.
| >
| > M> I tried something called sharpziplib but it doesn't support >2Gb
| > M> files -
| > M> they are working on it however but I cannot wait...
| >
| > Did you try GZipStream?
| > (
http://msdn2.microsoft.com/en-us/library/system.io.compression.gzipstream.aspx )
| > You must be using .NET 2.0 to work with it.
| >
| > --
| > Regards, Vadym Stetsyak
| > www: http://vadmyst.blogspot
|
| Looks like GZip wouldn't work either, from the 'Remarks' :
|
| "This class cannot be used to compress files larger than 4 GB."

Which is wrong too, actually the limit is 2GB, that's the max.size of an
object in .NET and the GZIP library needs the complete file to be read in a
single byte array (which obviously is limitted to 2GB).
Click to expand...

Why does GZIP require the entire file to be in memory first? It is a sliding
window algorithm similar to LZW. In any case, implementation is via a
GZIPStream, which doesn't seem to require the entire file be loaded into
memory first, so I believe 4GB is indeed the limit (which is the limit of
addressable memory with a 32-bit pointer).
 
| >
| > | > |
| > |
| > | "Vadym Stetsyak" wrote:
| > |
| > | > Hello, MrNobody!
| > | >
| > | > M> I need to zip up an 8Gb file using C# - do you know of any
freeware
| > | > M> libraries
| > | > M> I can use to accomplish this? I guess it needs to be Zip64 to
support
| > | > M> such
| > | > M> file sizes.
| > | >
| > | > M> I tried something called sharpziplib but it doesn't support >2Gb
| > | > M> files -
| > | > M> they are working on it however but I cannot wait...
| > | >
| > | > Did you try GZipStream?
| > | > (
| >
http://msdn2.microsoft.com/en-us/library/system.io.compression.gzipstream.aspx )
| > | > You must be using .NET 2.0 to work with it.
| > | >
| > | > --
| > | > Regards, Vadym Stetsyak
| > | > www: http://vadmyst.blogspot
| > |
| > | Looks like GZip wouldn't work either, from the 'Remarks' :
| > |
| > | "This class cannot be used to compress files larger than 4 GB."
| >
| > Which is wrong too, actually the limit is 2GB, that's the max.size of an
| > object in .NET and the GZIP library needs the complete file to be read
in a
| > single byte array (which obviously is limitted to 2GB).
| >
|
| Why does GZIP require the entire file to be in memory first? It is a
sliding
| window algorithm similar to LZW. In any case, implementation is via a
| GZIPStream, which doesn't seem to require the entire file be loaded into
| memory first, so I believe 4GB is indeed the limit (which is the limit of
| addressable memory with a 32-bit pointer).
|


The limit is indeed 4GB, but it has nothing to do with the addressable
memory, this comes from the gzip RFC which requires the input file size to
be less than 4GB.
The input size is written to the compressed file as an unsigned int (module
2^32), the GZipStream implementation keeps an internal count of the number
of bytes read from the input stream and throws:
"System.IO.InvalidDataException: The gzip stream can't contain more than 4GB
data." when the number of bytes read exeeds 2^32.

Willy.
 
Willy Denoyette said:
|
| Why does GZIP require the entire file to be in memory first? It is a
sliding
| window algorithm similar to LZW. In any case, implementation is via a
| GZIPStream, which doesn't seem to require the entire file be loaded into
| memory first, so I believe 4GB is indeed the limit (which is the limit of
| addressable memory with a 32-bit pointer).
|


The limit is indeed 4GB, but it has nothing to do with the addressable
memory, this comes from the gzip RFC which requires the input file size to
be less than 4GB.
The input size is written to the compressed file as an unsigned int (module
2^32), the GZipStream implementation keeps an internal count of the number
of bytes read from the input stream and throws:
"System.IO.InvalidDataException: The gzip stream can't contain more than 4GB
data." when the number of bytes read exeeds 2^32.
Click to expand...

Agreed. I shouldn't have limitted it to addressable memory and instead
limitted it to the resolution of a 32-bit WORD. GZIP was designed originally
for 32-bit processing. The same reason that a 32-bit system is limitted to
4GB addressable memory is the reason that GZIP is limitted to 4GB file sizes.

As I indicated before, there is no reason that one can not compress a file
using GZIPStream as long as it is less than 4GB in length (it doesn't have to
be loaded into memory to compress using the stream).
 
Back
Top