Advice for compressing TIFFs

  • Thread starter Thread starter cubilcle281
  • Start date Start date
C

cubilcle281

Hi all,

I have been busy with my scanning project, having scanned 1200 slides
over the last 10 days.

Getting the files off my hard drive was at first easy - the 126 slides
are all about 42meg, so 100 per DVD is quite easy (and logical, as each
tray holds 50 slides). However, now I am up to 35mm slides which are
more problematic. At 2700dpi, 16bits per color, the slides are over 50
meg each.

I would like to keep the DVDs in line with the trays, so I would prefer
to archive 50 (1 tray) or 100 (2 trays) at a time. 1 tray at a time
would be a waste, so I would like to investigate ways to compress the
TIFFs so that I can fit 2 trays per slide.

NikonScan 3.1.2 (latest release compatible with the LS-2000 doesn't
seem to support LZW compression, so I would need to go through a
3rd-party program. 2 questions:

1. Am I likely to achieve any saving with LZW compression on
16-bit-per-channel scans? Given that the LS-2000 only does 12-bits per
channel, I figure there should be a reasonable amount of dead space

2. Since many of the scans are done, is there a program available that
will do the conversion in 16-bit. Since PSE doesn't support 16-bit, I
haven't even tried that, and shelling out for the full photoshop for
this would be a waste.

TIA,

C
 
1. Am I likely to achieve any saving with LZW compression on
16-bit-per-channel scans? Given that the LS-2000 only does 12-bits per
channel, I figure there should be a reasonable amount of dead space

2. Since many of the scans are done, is there a program available that
will do the conversion in 16-bit. Since PSE doesn't support 16-bit, I
haven't even tried that, and shelling out for the full photoshop for
this would be a waste.
Click to expand...


The best compression schemes are image-dependent
in terms of *how much* compression you'll end up with.
This is true of LZW as it is with JPG.

Why not just try it and see? In general I find LZW
compression of images to be hardly worth the trouble.

Here's what I just got from Photoshop:

TIF: 402.8 MB (4000 dpi, 16/48bit scan of 645 MF film)
LZW compressed TIF, same file: 331.8 MB

As for compressing 16-bit TIFs after the fact, there's
nothing to stop you from ZIPing them. Ie., WinZip
also implements LZW, as I recall -- and does so on
arbitrary file formats and sizes.


rafe b
www.terrapinphoto.com
 
Why not just try it and see? In general I find LZW
compression of images to be hardly worth the trouble.

Here's what I just got from Photoshop:

TIF: 402.8 MB (4000 dpi, 16/48bit scan of 645 MF film)
LZW compressed TIF, same file: 331.8 MB
Click to expand...


Hold the presses. I got that backwards.

The "compressed" version is 402.8 MB.
The uncompressed version is 331.8 MB.

TIF-LZW **expanded** the file by 21%.

I should have caught this earlier. An 8-bit scan
of 645 is generally 150 M, 16-bit is 300 M.
(4000 dpi.)

LZW backfired. I think I've seen this before.



rafe b
www.terrapinphoto.com
 
Your followup post notwithstanding, I tried ZIP files using
powerarchiver, and saw a saving of only 3-5%. The other issue is that
the zip files did not like olding 5GB of data (at least, using
PowerArchiver).

I could always try individual zip files, but I need somewhere in the
region of 20% reduction in size to make things fit.

Is there a version of TIFF that supports 12-bit files, since that is
what the LS-2000 is actually producing?

C
 
cubilcle281 said:
Your followup post notwithstanding, I tried ZIP files using
powerarchiver, and saw a saving of only 3-5%. The other issue is that
the zip files did not like olding 5GB of data (at least, using
PowerArchiver).

I could always try individual zip files, but I need somewhere in the
region of 20% reduction in size to make things fit.

Is there a version of TIFF that supports 12-bit files, since that is
what the LS-2000 is actually producing?
Click to expand...

There is not a 1:1 relationship between the number of bits the scanner
sensor produces and the number of bits in a TIFF file. The scanner
sensor is linear; the data in image files is not. That is how 8-bit
JPEGS (for example) manage to cover a theoretical contrast range of
1:200000 - far more than any screen or paper can reproduce. A 12-bit
scanner will manage a maximum contrast range of 1:4000 (which is still
enough).

8-bit images are not so great if you try to make large changes to the
brightness or contrast. As long as you get the exposure about right
when doing the original scan, 8-bit images are fine. I have settled on
using JPEGs with the quality set to "high".

But don't take my word for it - make a few test prints to convince
yourself of the quality you need. The trial version of Picture Window
Pro will let you play with 16-bit files for 30 days.

-Tim
 
2. Since many of the scans are done, is there a program available that
will do the conversion in 16-bit. Since PSE doesn't support 16-bit, I
haven't even tried that, and shelling out for the full photoshop for
this would be a waste.
Click to expand...

My 2 cents:

It may be useless to you as you don't have or want to buy PS CS2 (I know why).
But I'll tell you what I did to compress my scans (234 MB) to reasonable size
(45 MB). I have to confess however that I am not completely convinced that one
day the files saved to dvd cannot be restored to TIF.

In PS CS2 there is a plugin for JPEG2000 (not the Adobe plugin, but one by
"fnord" <www.fnordware.com>. It is able to compress TIF lossless to J2K, and a
test showed that 50 MB was reduced to <38 MB. The "fnord" company may have
other plugins that are useful, I did not check it recently. They also offer
very little help if you are in trouble!

If you allow some loss, you can go much further. I have chosen for a lossy
compression. Tests showed to my satisfaction that I could not measure any
significant loss of quality going from 234 MB to 45 MB.

There is one catch. Can you read them back into PS? For some time I had severe
troubles doing just that. By sheer luck I found that if I had the
"ScriptingSupport.8li" as well as the "ScriptListener.8li" plugins installed,
there was no difficulty any more in reading back the J2K files from DVD. The
reason for this is totally unclear to me.

Will the method using J2K compression remain usable? is my big question. Or
should I go back and store the scans as TIF?
Greetings, Alex
 
Your followup post notwithstanding, I tried ZIP files using
powerarchiver, and saw a saving of only 3-5%. The other issue is that
the zip files did not like olding 5GB of data (at least, using
PowerArchiver).

I could always try individual zip files, but I need somewhere in the
region of 20% reduction in size to make things fit.

Is there a version of TIFF that supports 12-bit files, since that is
what the LS-2000 is actually producing?
Click to expand...


The TIF file standard fully supports 16-bit images,
layers, alpha masks, profile tags, etc.

Sharp images (eg film scans) do not compress
well from LZW compression, in my experience.

Have you tried PNG file format? Lossless, with
good compression, supports 16-bit. Better
compression than LZW anyway.


rafe b
www.terrapinphoto.com
 
cubilcle281 said:
Hi all,

I have been busy with my scanning project, having scanned 1200 slides
over the last 10 days.

Getting the files off my hard drive was at first easy - the 126 slides
are all about 42meg, so 100 per DVD is quite easy (and logical, as each
tray holds 50 slides). However, now I am up to 35mm slides which are
more problematic. At 2700dpi, 16bits per color, the slides are over 50
meg each.

I would like to keep the DVDs in line with the trays, so I would prefer
to archive 50 (1 tray) or 100 (2 trays) at a time. 1 tray at a time
would be a waste, so I would like to investigate ways to compress the
TIFFs so that I can fit 2 trays per slide.

NikonScan 3.1.2 (latest release compatible with the LS-2000 doesn't
seem to support LZW compression, so I would need to go through a
3rd-party program. 2 questions:

1. Am I likely to achieve any saving with LZW compression on
16-bit-per-channel scans? Given that the LS-2000 only does 12-bits per
channel, I figure there should be a reasonable amount of dead space

2. Since many of the scans are done, is there a program available that
will do the conversion in 16-bit. Since PSE doesn't support 16-bit, I
haven't even tried that, and shelling out for the full photoshop for
this would be a waste.

TIA,

C
Click to expand...
I would take a hard look at whether you need 16 bits/color or not.
Convert some of your 16 bit images to 8 can see if you can tell the
difference. Bring up the shadow detail and see if there seems to be
more noise in the 8 bit version then the 16. I have not seen many
images that benefit from having more then 8 bits. Some of this will
depend on the color space you use and how well the black point is set,
so you need to try on your scans and see if the 16 bits are needed in
your case.

BTW PSE 3 and above allow you to work with the 16 bit / color images
and is pretty cheap.

While you are at it try converting to 8 bit/color and then saving as a
quality 12 jpeg. For a very clean image a jpeg image will loss a bit
of the shadow detail that a 8 bit /color tiff will have, but most image
have so much noise in the shadow detail that this is not an issue.

Scott
 
SNIP
1. Am I likely to achieve any saving with LZW compression on
16-bit-per-channel scans? Given that the LS-2000 only does
12-bits per channel, I figure there should be a reasonable
amount of dead space
Click to expand...

There can be electronic noise in those lesser significant bits or,
depending on the software, the data may be stretched/scaled to fill
the 16-bits/channel.
16-b/ch data therefore compresses poorly, in fact it may sometimes
increase the file size.

Bart
 
I'd:

1. Stick with uncompressed 16 bit tiff. LZW is laughable, in my
experience also, it did "negative" compression. The other formats are
risky. I too have problems openings jpeg2000.

2. Forget about having some constant number to a disk. Just jam as many
in as will fit, and keep a log file stating first and and last file on
each disk, to facilitate finding them later.
 
SNIP

There can be electronic noise in those lesser significant bits or,
depending on the software, the data may be stretched/scaled to fill
the 16-bits/channel.
16-b/ch data therefore compresses poorly, in fact it may sometimes
increase the file size.

Bart
Click to expand...


That is exactly what I observed (and reported) about about
two posts back. LZW compression of a 16-bit TIF actually
*increased* the file size about 20%.

PNG seems to do better, and handles 16-bit files. I got
50% compression from PNG on a 122 Mbyte 35mm scan.


rafe b
www.terrapinphoto.com
 
Raphael Bustin said:
That is exactly what I observed (and reported) about about
two posts back. LZW compression of a 16-bit TIF actually
*increased* the file size about 20%.
Click to expand...

Compressing a 16-bit image file image file requires understanding that units
of three 16-bit values specify a color/intensity level on an XY plane. At
that point, "compression" consists more of redescribing the image in a more
efficient manner. Since real images consist of areas that change smoothly,
it takes a lot less data to describe those smoothly changing areas as sums
of sine (cosine, actually) waves than it does as a bitmap.

If LZW just looks at the data as a sequence of bytes, there's no way it can
find the "degeneracy" in the data.
PNG seems to do better, and handles 16-bit files. I got
50% compression from PNG on a 122 Mbyte 35mm scan.
Click to expand...

I've been largely ignoring PNG, but it seems to be exactly the right thing
for people who believe that every bit of their 16-bit scans is valid
information. Since it's an image format, it can apply intelligent
compression algorithms.

Personally, I think that 16-bit storage of film scans is largely meaningless
because of grain noise and the gross softness of the images, but that 16-bit
scanning makes some amount of sense. So the sensible sequence would be scan
in 16 bits, apply noise reduction, set black and white points, adjust
contrast and colors roughly, and save (and archive) as an 8-bit best quality
jpeg. You should get a factor of three compression from best quality jpeg if
you apply light noise reduction before saving. That's one byte per
megapixel, as opposed to three bytes per MP for compressed 16-bit PNGs.

In my TIFF vs. best quality JPEG tests, what I observed was that the only
pixels that got changed by jpeg compression were ones at extremely high
contrast transitions. Since 4000 dpi scans contain no such transitions
(edges are smeared over 4 to 6 pixels), best quality jpeg is essentially
lossless for scanned images.

David J. Littleboy
Tokyo, Japan
 
Personally, I think that 16-bit storage of film scans is largely meaningless
because of grain noise and the gross softness of the images, but that 16-bit
scanning makes some amount of sense. So the sensible sequence would be scan
in 16 bits, apply noise reduction, set black and white points, adjust
contrast and colors roughly, and save (and archive) as an 8-bit best quality
jpeg. You should get a factor of three compression from best quality jpeg if
you apply light noise reduction before saving. That's one byte per
megapixel, as opposed to three bytes per MP for compressed 16-bit PNGs.

In my TIFF vs. best quality JPEG tests, what I observed was that the only
pixels that got changed by jpeg compression were ones at extremely high
contrast transitions. Since 4000 dpi scans contain no such transitions
(edges are smeared over 4 to 6 pixels), best quality jpeg is essentially
lossless for scanned images.
Click to expand...


These are my observations almost exactly.

I can measure losses even from best-quality JPG conversion,
but I cannot see them, try as I may.

As you know, the two "Perfect Scans" on my scan snippets
site are there in part to show just how good JPG can be.
You probably remember all the same tired arguments about
JPG on r.p.e.m-f back when I first announced the "scan
snippets" site.

Also agree entirely with your approach to bit-depth at various
points in image processing. Extra bit-depth comes in handy
*during* tonal moves, but the end-result can be expressed
in 8-bit, almost always -- at least with the output devices that
are available to me (and this includes LightJet.)

Dan Margulis seems to suggest that Photoshop has clever
algorithms for introducing noise when truncating or rounding
files from N bits down to 8. This apparently reduces or
eliminates posterization. [This may simply boil down to
a bit of dithering..]

Anyway, once I'm storing a file in its final, corrected
state (as opposed to a "raw" scan) eight bits is really
enough. Hard to justify more than that on any hard
evidence. What print method holds more than 2^8
tones in any color or in pure luminosity?


rafe b
www.terrapinphoto.com
scan snippets
www.terrapinphoto.com/jmdavis
 
OK, so PNG might be a contender.

Next question then is how do I convert from TIFF to PNG? I would like
to automate this also. None of my software does 16-bits/channel
(remember, no Photoshop). Does Image Magick support 16-bit/channel
TIFFs and do TIFF to PNG in 16 bits/channel? I figure I can use my
limited knowledge of Perl I can produce a command stream, but from the
documentation I read on the site I couldn't tell whether
16-bits/channel was supported.

C
 
That is exactly what I observed (and reported) about about
two posts back. LZW compression of a 16-bit TIF actually
*increased* the file size about 20%.

PNG seems to do better, and handles 16-bit files. I got
50% compression from PNG on a 122 Mbyte 35mm scan.


rafe b
www.terrapinphoto.com
Click to expand...

Rafe,

I have promoted PNG for storage some years ago but due to the
not so good support in commercial applications it is a bit
tricky. At that time even Photoshop gave some some problems.
Qimage showed problems too then. In open source software the
support is usually better. I returned to Tiff after that but
could give it a try again.

The PNG classes are alright and include everything you could
wish including alpha layers, ICC profile embedding, greyscale
etc but it is a risk when the software doesn't use the right
conversions.

Ernst

--

--
Ernst Dinkla


www.pigment-print.com
( unvollendet )
 
Raphael said:
That is exactly what I observed (and reported) about about
two posts back. LZW compression of a 16-bit TIF actually
*increased* the file size about 20%.

PNG seems to do better, and handles 16-bit files. I got
50% compression from PNG on a 122 Mbyte 35mm scan.


rafe b
www.terrapinphoto.com
Click to expand...

While LZW compression adds :-) 130 MB to a 462 MB 16 bit TIFF
6x6 scan, PNG made it not more than 47 MB smaller with 415 MB.
10% reduction. Still better than zipping the file which
reduced it to 441 MB and Photoshop's TIFF zip compression
makes it 447 MB. The last is supported in Qimage as is TIFF
LZW compression and PNG. PNG is slow in expanding though.

This thread at least made me aware that I should not use LZW
as a default.

Ernst



--

--
Ernst Dinkla


www.pigment-print.com
( unvollendet )
 
Will the method using J2K compression remain usable? is my big question. Or
should I go back and store the scans as TIF?
Click to expand...

Unfortunately, J2K doesn't seem to have been adopted widely. It does
show up in various software but most people seem to continue with
plain-vanilla JPG.

Basically, as long as you keep the software which can convert J2K back
to some other format you should be "future proof".

One other thing. All compressed formats are much more susceptible to
data corruption! If a single bit fails in a compressed file some
programs may refuse to read the whole file or (if you're lucky) only
data following the failed bit. Either way, you will virtually lose the
image.

On the other hand, if you use an uncompressed format then lots of data
can fail and you can still recover the image, perhaps (conceptually)
treating failed bits as "digital dust" or "digital scratches".

So, considering all that and the constantly falling price of media I
don't think any compression is really worth the trouble for the
purpose of permanent archiving.

Don.
 
cubilcle281 said:
OK, so PNG might be a contender.

Next question then is how do I convert from TIFF to PNG? I would like
to automate this also. None of my software does 16-bits/channel
(remember, no Photoshop). Does Image Magick support 16-bit/channel
TIFFs and do TIFF to PNG in 16 bits/channel? I figure I can use my
limited knowledge of Perl I can produce a command stream, but from the
documentation I read on the site I couldn't tell whether
16-bits/channel was supported.
Click to expand...


There's an ImageMagic user forum, I believe, where
you could post that question directly.

Click the "Community" link at the top of the ImageMagick
home page:

<<http://www.imagemagick.org>>

Nobody can "vouch" for the viability of PNG or any
other format. You understand that, right? Ultimately,
we're all just guessing and taking our chances.


rafe b
www.terrapinphoto.com
 
cubilcle281 said:
OK, so PNG might be a contender.

Next question then is how do I convert from TIFF to PNG? I would like
to automate this also. None of my software does 16-bits/channel
(remember, no Photoshop). Does Image Magick support 16-bit/channel
TIFFs and do TIFF to PNG in 16 bits/channel? I figure I can use my
limited knowledge of Perl I can produce a command stream, but from the
documentation I read on the site I couldn't tell whether
16-bits/channel was supported.
Click to expand...

The newer versions of Photoshop elements handle 16 bit/ color tiff and
png files just fine.
I did a test with converting a tiff to a 16/color png, the size of the
png was about 36% the size of the tiff. It surprised me that it would
be that much, this is a pretty clean image so that might help. I did
check and the full 16bits of data seems to all be there.

Scott
 
There's an ImageMagic user forum, I believe, where you could post that
question directly. http://www.imagemagick.org/
Click to expand...

IIRC, the last time I looked at ImageMagick for Windows, they had 4
different packages available, 2 that did 8-bits/channel and 2 that did
16-bits/channel. Pick the right one and use it, no problem.
Nobody can "vouch" for the viability of PNG or any other format. You
understand that, right? Ultimately, we're all just guessing and
taking our chances.
Click to expand...

Er... huh? There are a few image formats that are fully described and
have reference implementations available in C. TIFF, PNG, and JPEG are
all formats like that. libjpeg, libtiff, and libpng are all under a
BSD-ish license, so any programmer who's competent can build an app to
convert a raw bitmap <-> {TIFF,JPEG,PNG} pretty easily. So formats like
those are future-proof as long as competent programmers still exist.
(rafe b was probably thinking as a user, not a programmer, though.)

If you store images (or any data, really) in a format that's not Open,
you will eventually lose that data. People have been saying this for
the last 10 years, but the vast majority of users haven't grokked it
yet.
 
Back
Top