Vuescan: dark frame subtraction

  • Thread starter Thread starter Fernando
  • Start date Start date
F

Fernando

Just to say that I started working on it (please cfr. the "VueSCAM"
thread for details), but I seem to have encountered a curious bug in
the ImageMagick libraries I'm using, that leads to image corruption.

I'm filing a bug report, hoping this will addressed in a reasonable
time; in the meanwhile, I had to put the project on hold. :(

Or... does anyone know a capable, royalty-free and easy to use C
graphic library that allows reading and writing 16bits/channel
RGB/RGBA TIFF files?

Libtiff does not apply: not easy to use (on the contrary), and does
not work with 16bits/channel unless going on lower-level functions
(and the higher-level ones are complicated enough!).

Thanks,

Fernando
 
Or... does anyone know a capable, royalty-free and easy to use C
graphic library that allows reading and writing 16bits/channel
RGB/RGBA TIFF files?

Why don't you just access the files directly? TIFF format has been
well documented and parsing the headers is elementary.

Don.
 
Because I find it extremely annoying to do even using LibTIFF; let
alone going directly through the file.
The TIFF is probably the most complicated graphic file format ever
written...!

Of course, if you happen to have a TIFF reading/parsing C routine that
you feel OK to share, I'm all ears! :-)

Fernando
 
* Fernando said:
Not yet... I have only minimal spare time to dedicate to this utility,
but if the ImageMagick team won't fix the bug across the weekend, I'll
have to look elsewhere to read TIFF files. I have lots of scans to do,
and would appreciate having them streaks-free.
The problem is that even the TIFF 6.0 specifications document does not
mention 16 bits/channel, so I should go by trial & error, not exactly
fast... I have to learn how to read 48bit RGB and 64bit RGBA files at
the very least.

Try http://cinepaint.sf.net source. I know for sure that it reads 16b/c
TIFF, and is likely to have reading code well separated.
ImageMagick routines are extremely easy to use, powerful and flexible...
I really hope they will fix the bug. :(

If you know what the bug is, then why not fix it yourself :)
 
degrub said:
Have you taken a look at the Lcms code ?

Not yet... I have only minimal spare time to dedicate to this utility,
but if the ImageMagick team won't fix the bug across the weekend, I'll
have to look elsewhere to read TIFF files. I have lots of scans to do,
and would appreciate having them streaks-free.
The problem is that even the TIFF 6.0 specifications document does not
mention 16 bits/channel, so I should go by trial & error, not exactly
fast... I have to learn how to read 48bit RGB and 64bit RGBA files at
the very least.
ImageMagick routines are extremely easy to use, powerful and flexible...
I really hope they will fix the bug. :(
Sheez, I found a bug in a library while writing a workaround for a bug
in a software! Call me a lucky man! :D

Fernando
 
Fernando said:
The problem is that even the TIFF 6.0 specifications document does not
mention 16 bits/channel,

Oh it does, albeit indirectly. In fact, the specification mentions the
capability for up to 65535-bit/channel - a little OTT I feel. ;o)

The specification does not give an example of how a 16-bit/channel image
is constructed, but it is just an RGB image with 16-bits per channel -
only one set of parameters is different from the 8-bit RGB example it
does describe in detail.

BitsPerSample defines the number of bits in *each* sample - so can even
have different bit depth in each of the RGB if necessary. Some 16-bit
colour images use 5 bits red, 6 bits green and 5 bits blue, for example.
For 16-bit RGB, the BitsPerSample of each channel is 16.

SamplesPerPixel defines the number of channels, 3 for RGB, 4 for RGBI.

Hope that helps.
 
Pasi said:
Try http://cinepaint.sf.net source. I know for sure that it reads 16b/c
TIFF, and is likely to have reading code well separated.

Thanks, I'll look at it!
If you know what the bug is, then why not fix it yourself :)

If I knew where the bug was, I would already have fixed it. :)

At first it looked like a simple endianness problem (also to the
ImageMagick team), but it isn't that simple, so I gave up.
It does not occur on all images; it seems to occur only on (certain?)
Vuescan RAW files (in my experience at least), but I failed to
understand what is "strange" in those files.

Fernando
 
Kennedy said:
Oh it does, albeit indirectly. In fact, the specification mentions the
capability for up to 65535-bit/channel - a little OTT I feel. ;o)

Kennedy, could you point me to the proper document section? I failed to
find any hint about > 8 bits/channel in the RGB section of the 6.0
paper.. :(
The specification does not give an example of how a 16-bit/channel image
is constructed, but it is just an RGB image with 16-bits per channel -

Do you know if the 16-bits data are represented as a whole short
(16-bits) unsigned integer or as two separate bytes, and with which
byte-wise endianness? I've encountered all sorts of odd choices while
reading graphic files, so I no longer take anything for granted. :)

Thanks a lot!

Fernando

(I still hope ImageMagick people would fix the bug; this way I may have
an alpha-release of the utility for both Linux and Windows in a couple
hours at Saturday night when I'm back in town)
 
At first it looked like a simple endianness problem (also to the
ImageMagick team), but it isn't that simple, so I gave up.
It does not occur on all images; it seems to occur only on (certain?)
Vuescan RAW files (in my experience at least), but I failed to
understand what is "strange" in those files.

In case anyone is interested, I posted a before/after example of what
I'm talking about, here (6.5 MB zip):

http://gundam.srd.it/PhotoPages/temp/bugreport-imagesamples.zip

Infos:

The original file is a 48bits RGB Linear image; it was only resized
with Photoshop CS from the Vuescan RAW file (a 240MB beast).

The "cleaned" file was actually saved just after it was read: despite
the filename, I did not touch anything, just loaded and saved it.

If someone with a good TIFF knowledge can offer a clue about what's
unusual with the original file... ImageMagick can read and write other
TIFF images without problems, it shows this odd behaviour only with
those RAW scans. Of course Photoshop can read and write those RAW
scans just fine.

Seems like I won't be back at home this night, but only on Sunday
night, but I'll rush here then! :)

Thanks to all of you,

Fernando
 
Because I find it extremely annoying to do even using LibTIFF; let
alone going directly through the file.

The TIFF is probably the most complicated graphic file format ever
written...!

No way! Have you tried BITMAP? TIFF is real easy by comparison.
Of course, if you happen to have a TIFF reading/parsing C routine that
you feel OK to share, I'm all ears! :-)

No, I don't do C. I prefer real languages... ;o)

As others have mentioned you need to get the latest specs. 16-bit is
just an implicit parameter, nothing special per se.

Anyway, if you can wait, I expect to do some TIFF massaging in the
next couple of weeks and will be able to answer more concrete
questions then.

Don.
 
Fernando said:
Just to say that I started working on it (please cfr. the "VueSCAM"
thread for details), but I seem to have encountered a curious bug in
the ImageMagick libraries I'm using, that leads to image corruption

ImageMagick does dark frame subtraction on the commandline (see compose
command) as well as tiffcp (from the libtiff package). Did you try
that?
 
Fernando said:
Kennedy, could you point me to the proper document section? I failed to
find any hint about > 8 bits/channel in the RGB section of the 6.0
paper.. :(
The spec document I have is quite old (June 3 1992) but I believe is
still the most recent publicly available version.

In Section 8, titled "Baseline Field Reference Guide" at the top of page
29, the definition of BitsPerSample is given as:
"Number of bits per component.
Tag = 258 (102.H)
Type = SHORT
N = SamplesPerPixel
Note that this field allows a different number of bits per component for
each component corresponding to a pixel. For example, RGB color data
could use a different number of bits per component for each of the three
color planes. Most RGB files will have the same number of BitsPerSample
for each component. Even in this case, the writer must write all three
values."

On Page 15, under Types, SHORT is defined as a "16-bit (2-byte) unsigned
integer".

So, in theory, the each colour in the tiff format can be as much as
65535 bits - or as many as "1 followed by 19,728 zeros" levels. For an
RGB file that is "1 followed by almost 60thousand zeros" of potentially
different colours. Just a little OTT, no? ;-)
Do you know if the 16-bits data are represented as a whole short
(16-bits) unsigned integer or as two separate bytes,

It is defined as a short unsigned integer, but can obviously be read
from the file as two separate bytes if that is easier in the language
you are using. Providing you get the bytes in the right significant
order, it doesn't make any difference.
and with which byte-wise endianness?

A tiff file uses the same endian structure throughout the file, and this
is defined in the first two bytes of the file:
49h,49h is Ascii for I,I, abbreviating "Intel" who use little-endian
structures.
4dh,4dh is Ascii for M,M, abbreviating "Motorola" who use big-endian
structures.

The best bit is yet to come though! The next byte in a TIFF file is,
according to the specification, "An arbitrary but carefully chosen
number that further identifies the file as a TIFF file."

It is the answer to that ultimate question about life, the universe and
everything. The answer that Deep Thought, the second most powerful
computer in the universe, took seven and a half million years to compute
and the question that the white mice designed built and programmed
planet earth to ask, to which that was the answer: "What does it all add
up to: Life, The Universe and Everything?".

The number is, as every h2g2 listener, reader and/or watcher knows, is..

42 !!!

Douglas Adams lives!
Or at least had some fans at Aldus! ;-)

Seriously though, it is a testament to both the readability, structure
and flexibility of the original specification by Aldus (now Adobe) that
the tiff format remains the most popular non-lossy image file format in
use today, despite being essentially unchanged since its original public
release in 1987. Quite incredible really, given the changes in the
technologies that use it in that time.

The only limitation I think might require update soon is the size limit
of 2^32 bytes in total. With FAT16 and FAT32 filing systems this was OK
since they could only handle similarly maximum file sizes of 4Gb, but
it's getting a bit tight for some large, high resolution, multi-image
files now on NTFS and similar systems.

Well, in a couple years we'll be at the 20th anniversary of "TIFF" - I
wonder if Adobe will update the specification to overcome such
limitations then.

Still, for the moment, I think I'll sit here and enjoy a pan galactic
gargleblaster! ;-)
 
It is the answer to that ultimate question about life, the universe and
everything. The answer that Deep Thought, the second most powerful
computer in the universe, took seven and a half million years to compute
and the question that the white mice designed built and programmed
planet earth to ask, to which that was the answer: "What does it all add
up to: Life, The Universe and Everything?".

The number is, as every h2g2 listener, reader and/or watcher knows, is..

42 !!!

Douglas Adams lives!

Yes! ;o)

My very first article in a computer magazine had the phrase "the
answer to that ultimate question about life, the universe and
everything". It was prominent in the very first paragraph and the
editor picked up on it.

The article appeared on - you guessed it - page 42! ;o)
Still, for the moment, I think I'll sit here and enjoy a pan galactic
gargleblaster! ;-)

In the restaurant at the edge of the universe, no doubt! ;o)

Don.
 
ImageMagick does dark frame subtraction on the commandline (see compose
command) as well as tiffcp (from the libtiff package). Did you try
that?

Yes, but the problem lies in the need of averaging the dark frame,
first, in order to get rid of the random noise.
That said, the Compose command fails as well on those images ("strange
colors" output), see my other message... it's a problem shared across
the whole ImageMagick package.

I'm going to try the 6.2.0-6 now to see if they addressed the bug.

Fernando
 
I'm going to try the 6.2.0-6 now to see if they addressed the bug.

They did, apparently!
At least in the Unix source version.

Expect an alpha-release within a hour or so.

Fernando
 
For the adventurous guys :-)

http://gundam.srd.it/PhotoPages/software/dfsub-0.1.zip

I assume you already know about what do to (what a dark frame scan is,
how to work in linear space -or better, in RAW mode-, etc.).
If not, please read all this thread and also the "VueSCAM" thread.

=> You'll need the ImageMagick free runtime libraries: <=

http://www.imagemagick.net/www/download.html?


NOTE: DFSub is a command-line utility!

I needed an executable that was fast to code and easy to script, so
the choice.

You'll maybe run into some harmless "Unknown Field" messages then the
utility loads some TIFF files. Ignore them.

Enjoy, and let me know. Please note that the software is "as is",
without any type of warranty: use at your own risk...

I have very little spare time at the moment, but will try to improve
this small utility for better streaks suppression: contributions are
welcome.

It is free software released under the GNU Public License; C source
code is included, along with Windows and Linux binaries. Needless to
say, the Linux version is faster. :)

Fernando
 
For the adventurous guys :-)

http://gundam.srd.it/PhotoPages/software/dfsub-0.1.zip

I assume you already know about what do to (what a dark frame scan is,
how to work in linear space -or better, in RAW mode-, etc.).
If not, please read all this thread and also the "VueSCAM" thread.

=> You'll need the ImageMagick free runtime libraries: <=

http://www.imagemagick.net/www/download.html?


NOTE: DFSub is a command-line utility!

I needed an executable that was fast to code and easy to script, so
the choice.

You'll maybe run into some harmless "Unknown Field" messages then the
utility loads some TIFF files. Ignore them.

Enjoy, and let me know. Please note that the software is "as is",
without any type of warranty: use at your own risk...

I have very little spare time at the moment, but will try to improve
this small utility for better streaks suppression: contributions are
welcome.

It is free software released under the GNU Public License; C source
code is included, along with Windows and Linux binaries. Needless to
say, the Linux version is faster. :)

Fernando
 
Kennedy said:
So when will you post some results - some of us don't have scanners with
this problem but have an interest in how well the solution performs!
;-)

:)
First tests show the same result I posted last week (using the
Photoshop-filtered dark frame): still some faint streaks visible, but
no noise increase as it was the case in the very first attempt (without
averaging).
I'll do some more tests tonight, and will post some pics.
At a first glance, it looks like a simple dark frame subtraction can't
suppress all the streaking... but it could be related to the averaging
maybe: I average the dark frame across the whole frame (scan
direction), and maybe the streaks are not of constant intensity. So
the averaging process gives "streaks levels" that are maybe a bit too
low in a region and too high in another one.
Just a thought, I really didn't have the time to conduct any proper
test... time is so tight for me these days. :(

Fernando
 
Back
Top