Best scanning manager program?

  • Thread starter Thread starter T. Wise
  • Start date Start date
I've read that DVD's are not as archival as CD's. Delkin Devices
markets a CD, "eFilm Archival Gold," with the subtitle "The 300 Year
Disc." I'm converting my archival CD's to this brand. It's more
expensive, but I figure my images are worth it. Any comments?

No one knows for sure how long even the best optical disks will last.
Accelerated life testing has been extrapolated to show several disks
*should* be very long life, but no one knows for sure. Some have
turned out to be dismally shy of what had been predicted. For long
life you go with the best you can find and treat them nice.

Archived data can be rendered useless due to a number of causes of
which the normal storage life of the disk is only one. Gold, as in
the original Kodak disks was listed as potentially 100 years. It's
far more likely the medium will become obsolete long before it
actually fails.

For me, CDs are already obsolete. I shoot about 80 gigs worth of
digital images a year, or have been. I've also been scanning 35mm
slides and negatives which generate 60 to 128 meg files for each image
at 4000 dpi. For me, CDs are just too small to be practical. I still
have a lot of archiving to do and using two DVDs, with one local and
one remote I can use a lot of them in a hurry. I also have over 3
terabytes of on-line storage using 250 and 300 gig Ultra ATA HDs in
USB-2 enclosures. Two external drives with two or three internal is a
lot of storage when you take into account 4 computers. I'm running a
gigabit network using Cat5e cable, but would prefer something a tad
faster.
BTW, the OP asked about scanner managers & we've long ago wandered into
other territories (dual processors?), which good conversations
naturally do. But, at some point, when the conversation shifts to a
different topic (even a different group?) the moderator, or someone,

Moderator? No such thing. It's just up to the users to either guide
the conversation back, or start a new thread.

Roger Halstead (K8RI & ARRL life member)
(N833R, S# CD-2 Worlds oldest Debonair)
www.rogerhalstead.com
 
As I said previously, I don't quite agree on this, *as long as* the
noise is *only* applied so that it fills in the gaps, *without* touching
the existing data.

That's because this noise addition is lossless, and the "original" image
without noise can be precisely reconstructed, as long as one also has
its histogram.

The problem is that's just physically impossible. There's only a fixed
amount of pixels in an image, and the histogram shows them all.

So, if you want to "fill in the gaps" those pixels you need to fill in
the gaps have to come from somewhere. And that "somewhere" is regular
data i.e. pixels from different parts of the image. So you'll be
"robbing Peter to pay Paul" as we say in English.
It's not "pure data", that's for sure, but the "pure data" can be
reconstructed (with the "save the histogram" caveat), and the result
tends to be more pleasing to the eye, since it hides posterization.

The closest thing is to "redistribute" i.e. "reshuffle" the pixels all
over again but that's "corruption" by definition. You're really
"repainting" the image.

One such way is to scale the image up and down as I mentioned last
time. Using "bicubic interpolation" will add the least corruption, but
there will be corruption nevertheless. So a much better way is
described below...
Indeed we may.
And you're not wrong saying that it is mathematically impossible;
however, the mathematical impossibility falls if we assume that the
histogram of the original image is not discarded. If it's kept -- and
the noise is applied in the correct way, i.e. *only* filling the gaps --
there should be no loss of information at all.

As I explained above, that's just impossible because those pixels have
to come from somewhere.

If there is posterization, the only real solution is to "oversample"
i.e. go to a higher bit depth, scan and then reduce bit depth. This
will produce a smooth histogram at this lower bit depth. But
technically such an image will contain less data that the original
scan. And since - due to nature of computers - bit depth goes in
increments of 8, such reduction is very drastic! Going from 16-bit to
8-bit, for example, squeezes 256 different values into a single value.
Yes, that will produce a smooth histogram, but the price of reduced
bit depth is very high unless it's done as the very last step i.e.,:

The best overall approach (assuming one wants maximum quality) is to
use the highest bit depth available to scan e.g. 16-bit, do all the
edits in 16-bit and then reduce to 8-bit before outputting to printer,
or conversion to JPG for display.

Don.
 
Just to clarify, Filmget and Vuescan both scan at the scanner's maximum
bit depth. Filmget doesn't do a RAW gamma 1.0 scan as far as I am
aware. I have concluded that if any color corrections are enabled in
Filmget, they are performed at 8-bit precision, judging by the gaps in
the resultant histogram for the 16-bit file.

Stop right there! Several problems already...

First of all, to do any meaningful testing you have to *disable* all
image processing!!!

Secondly, what 16-bit histogram program are you using?
So if you accept that this is "objective," it still doesn't answer
which program gave me a better file, because of the visual flaws I
documented with Filmget's IR cleaning. I don't know of any objective
tool that measures noise patterns, so how can you objectively judge
this?

By using *objective* tests! The above "tests" you performed are
totally meaningless and pointless, as already explained.
I want to know from you Don, what are the limitations to what you can
measure, if you throw out any sort of visual judgment as subjective?

That question doesn't make *any* sense whatsoever!

What you have to grasp, Roger, is that *objective tests* and
*subjective feelings* are NOT related!!! No connection! Nada! Null.
Zero! Zip!

As long as you keep struggling to make a connection between them
you'll get nowhere except keep asking meaningless questions and
running around in circles.
How do you even conclude there is noise without visual judgment?

Oh, come on Roger. That's just silly beyond belief!

It's like saying "How do you tell which piece of string is longer
without guessing?".

!!! ===> You take a ruler and you ***measure*** them!!!! You
***don't*** look at them and ***guess***!!!! <=== !!!

Sheesh, I'm running out of exclamation marks... ;o)

You're tying yourself in knots, Roger, trying to prove that "guessing"
(the length) is not only "better" than measuring, but it's more
"accurate"! That's just sheer nonsense!

Really, Roger, that's beyond elementary and I'm beginning to wonder
how much point is there to this thread anymore.

The rest is just too silly to respond to.

Until you grasp these axiomatic basics I can write until I'm blue in
the face and it won't make any difference.

So, please go back and re-read the thread until all that's cleared up.
You're just jumping from one irrelevant point to another and then
going back and starting all over again.

I've answered all you points several times already!!!

I understand all this may seem confusing and frustrating to you but
this is getting quite pointless. You're going backward, not forward.

So, if you really want to get to the bottom of this, please go back
and re-read the thread. But *read* and *think*, don't just emotionally
react! It's all there. Explained *multiple* times!

Don.
 
Don said:
[snip: histogram smoothing by adding noise]
It's not "pure data", that's for sure, but the "pure data" can be
reconstructed (with the "save the histogram" caveat), and the result
tends to be more pleasing to the eye, since it hides posterization.

The closest thing is to "redistribute" i.e. "reshuffle" the pixels all
over again but that's "corruption" by definition. You're really
"repainting" the image.

Sorry, I still stand by what I said.

And what I said was: adding noise to an image whose histogram has
"gaps", so that such gaps are filled, is a reversible operation provided
that:
- the original histogram is not discarded
- noise is applied in an appropriate way
- the algorithm used is known (but this is obvious, and holds for every
"reversible", or "lossless", operation)

Instead of going to great lengths trying to explain why I still think
so, I'll provide an example.
If you download http://ljl.150m.com/scans/snoise.tar.gz , you will find
a small (and ugly) C program that does what I'm talking about.

There is a Linux executable included; sorry, no Windows compiler
available at the moment.

You should get a posterized ("gapped histogram") grayscale image and
save it to "raw" format (Photoshop's raw is fine).
Or you can use the included "test.raw".

Using "test.raw", run

../snoise -add 300 465 test.raw test_noise.raw histogram.txt

Now load the output image in Photoshop or something: you'll see that
noise has been added, and that the histogram has no gaps anymore. The
histogram is still all but "smooth" (you'll see what I mean), but there
are no gaps in any case. I'm sure a decent algorithm would make it
smooth for real.

Now run

../snoise -rem 300 465 test_noise.raw test_denoised.raw histogram.txt

You can check with Photoshop that "test_denoised.raw" is perfectly
identical to the original "test_noise.raw" (save bugs, but it works with
the included test image).


Just in case, the general syntax is

../snoise [-add|-rem] ImageWidth ImageHeight InputImage.raw
OutputImage.raw HistogramFile.txt

by LjL
(e-mail address removed)
 
Sorry, I still stand by what I said.

And what I said was: adding noise to an image whose histogram has
"gaps", so that such gaps are filled, is a reversible operation provided
that:
- the original histogram is not discarded
- noise is applied in an appropriate way
- the algorithm used is known (but this is obvious, and holds for every
"reversible", or "lossless", operation)

The problem is that - even though I myself am a devout "histogram
worshipper" - it's important to keep in mind that a histogram only
contains a count of pixels but not where those pixels are in the
image. It's possible to have two totally different images produce
identical histograms!

Here a little test to try:

Create an image, split it in two and then fill one half with white,
and the other with black.

Secondly, create another image but divide it into 4 equal parts (e.g.
4 squares) and fill two with white and two with black.

The histograms from both of those images will be identical. However,
the border between black and white in the second image will be twice
as long as the border in the first image!

Therefore, after removing posterization the resulting gray border in
the second image will be twice as long resulting in two totally
different histograms.

Another extreme case is to create an image where all pixels alternate
in a chessboard pattern.
Instead of going to great lengths trying to explain why I still think
so, I'll provide an example.
If you download http://ljl.150m.com/scans/snoise.tar.gz , you will find
a small (and ugly) C program that does what I'm talking about.

There is a Linux executable included; sorry, no Windows compiler
available at the moment.

I downloaded the code but haven't run it. However, by adding random
noise you are *uncontrollably* corrupting the image and corrupting it
much more than if you were to smooth it out with a method based on
image content like interpolation. Bicubic interpolation, for example,
does not only use two neighboring pixels, but four!

But even that method has limitations. A prime example is applying it
to the above "pixel chessboard" image. That's because, in the end, any
method to remove posterization is corruption.

The best (in the sense of data integrity) is still to scan in higher
bit depth and then reduce after all the image editing has been done.

Don.
 
Don said:
On Sat, 24 Sep 2005 02:59:41 +0200, "Lorenzo J. Lucchini"

[snip]
Instead of going to great lengths trying to explain why I still think
so, I'll provide an example.
If you download http://ljl.150m.com/scans/snoise.tar.gz , you will find
a small (and ugly) C program that does what I'm talking about.

There is a Linux executable included; sorry, no Windows compiler
available at the moment.

I downloaded the code but haven't run it. However, by adding random
noise you are *uncontrollably* corrupting the image and corrupting it
much more than if you were to smooth it out with a method based on
image content like interpolation. Bicubic interpolation, for example,
does not only use two neighboring pixels, but four!

It all depends on what you mean by "corruption", and by "uncontrollably".
Sure, as I'm adding noise to the data, I'm "corrupting" them. Also, the
final result would possibly more pleasing with some interpolation.

However, right now I wasn't focusing on the pleasantness of the final
result, but just on its reversibility.

The method I quickly implemented does not give results that are
exceptionally pleasing to the eye; on the other hand, it is completely
reversibly.

So while I can understand "corruption", from an aesthetical point of
view, I cannot understand "uncontrollability".

Just feed the program the "corrupted" image together with the original
histogram, and you'll get back precisely the original image.
*Not just* one having the original histogram, but the *same* image.
But even that method has limitations. A prime example is applying it
to the above "pixel chessboard" image. That's because, in the end, any
method to remove posterization is corruption.

Please note that I'm not claiming to *remove posterization*.
That's also why I snipped what you wrote at the beginning of the
article: with your black-and-white example images, applying "my method"
wouldn't result in removing posterization, by any stretch of imagination.

It will just add noise, the result perhaps not being very good-looking,
but reversible.

http://ljl.150m.com/scans/bw_orig.tif - a B/W image like one you suggest
http://ljl.150m.com/scans/bw_noise.tif - that image with noise applied
http://ljl.150m.com/scans/bw_back.tif - that image with noise removed
(change the extensions to .gif if you prefer)

You can see that the third image is identical to the first (not just
that the *histograms* are identical).

That's all I wanted to demonstrate: that noise can be applied in a way
that is reversible, as long as the histogram is considered a legitimate
part of the data needed for reversing.

By the way, the new http://ljl.150m.com/scans/snoise.tar.gz I've just
uploaded contains also a Windows executable.

by LjL
(e-mail address removed)
 
No. Speedup is only partially dependant on software's multiple processor
awareness.

That would be why a recent test found that changing an AMD 64 for an
AMD dual showed a 95% increase in 3D Max speed then...

Sort of blows your hypothesis out of the water.

Anyone with any sense knows that RAM, tasks, etc are going to make a
difference to any speed increases but dual processing for dual
processor aware apps makes a far greater difference than you think.

--

Hecate - The Real One
(e-mail address removed)
Fashion: Buying things you don't need, with money
you don't have, to impress people you don't like...
 
Hecate,

I see that you and your company do a lot of toil and go to a lot of
trouble in your backup approach. Perhaps someone's brain was bubbling
over with ideas that have the effect of doubling the workload.

Nope. When losing images means losing money you'll think differently.
For me, as a home network user, high quality tape has the advantages
of simplicity and compactness.
Tape is incredibly unreliable - for example, if your tape drive dies
and you buy an identical replacement it's not unknown for the new
drive to be unable to read the tape.

Let me ask you - have you ever performed (or tried to) a full restore
from tape?

--

Hecate - The Real One
(e-mail address removed)
Fashion: Buying things you don't need, with money
you don't have, to impress people you don't like...
 
I've read that DVD's are not as archival as CD's. Delkin Devices
markets a CD, "eFilm Archival Gold," with the subtitle "The 300 Year
Disc." I'm converting my archival CD's to this brand. It's more
expensive, but I figure my images are worth it. Any comments?

Generally, all the information I've seen is directly opposed to what
you've heard. And any method of backup is only as good as the devices
which have to restore the data. Consequently, I would never suggest
backing up on old technology, particularly when this technology is
being rapidly replaced. It won't be too long before we have to start
thinking about moving from DVD to HD-DVD as a storage method. By then,
you'll be lucky to be able to buy a drive which reads CDs that isn't
in a consumer music player.
BTW, the OP asked about scanner managers & we've long ago wandered into
other territories (dual processors?), which good conversations
naturally do. But, at some point, when the conversation shifts to a
different topic (even a different group?) the moderator, or someone,
should open a new topic in this or a different group and
cross-reference to the new topic/group.

This is Usenet. You don't get moderators. Threads drift. I'm afraid
that's life. (And it's what makes Usenet groups far more interesting
than Web based fora.

--

Hecate - The Real One
(e-mail address removed)
Fashion: Buying things you don't need, with money
you don't have, to impress people you don't like...
 
Generally, all the information I've seen is directly opposed to what
you've heard.

This is for you and Roger. Archiving Digital is a new group addressing
the archiving issue. It has few respondents, now, but anyway, there it
is. Another group, rec.photo.digital, is older & contains posts
questioning suitability of DVD for image archiving. There is another
group, which I can't provide a title for, devoted to archiving images
by libraries, which are big on archiving.

What I'm saying is, people interested in archiving digital images
should seek other groups dealing with it.
 
This is for you and Roger. Archiving Digital is a new group addressing
the archiving issue. It has few respondents, now, but anyway, there it
is. Another group, rec.photo.digital, is older & contains posts

You will probably find my name heavily involved in the discussion on
RPD. The question and discussion resurfaces over there every few
months.

I'm glad to hear they are starting up some groups devoted to archiving
data, but I'm afraid that the questions and discussions will most
likely stay with the photography and scanning groups where the
interest lies. It's difficult to get discussions pertaining to what
people are doing in a group to move to a specific group even if it is
devoted to that item. Sorta like RPD and RPD.slr-systems. There is
still more SLR stuff on RPD than systems many months after the
creation of the new group. Enough if you are looking for digital slr
stuff you still need to read RPD in addition to the digital
slr-systems.

Nothing shows up on a search of the newsgroup index using
archiving.digital. Actually, only one group shows up under archiving
and that is of no use. Nothing under digital or archive pertaining to
data archiving either.
OTOH, I'm on Charter and the newsgroup service they use ... well...
stinks is the most polite way I can put it. <:-)) It's slow, it's
throttled, connections are limited, and it's been my experience it
takes forever to get something added even if you have the correct and
complete name.

It looks like I'm going to have to find one of the subscription news
groups with a fast return and logical layout.
questioning suitability of DVD for image archiving. There is another
group, which I can't provide a title for, devoted to archiving images
by libraries, which are big on archiving.

What I'm saying is, people interested in archiving digital images
should seek other groups dealing with it.

I would if I could, but I can't.
Absolutely zip on Charter.

Roger Halstead (K8RI & ARRL life member)
(N833R, S# CD-2 Worlds oldest Debonair)
www.rogerhalstead.com
 
That would be why a recent test found that changing an AMD 64 for an
AMD dual showed a 95% increase in 3D Max speed then...

Sort of blows your hypothesis out of the water.

Also why Intel has had to rush dual core CPUs into the market. "Intel
Inside" but "AMD Leading" these days.
Anyone with any sense knows that RAM, tasks, etc are going to make a
difference to any speed increases but dual processing for dual
processor aware apps makes a far greater difference than you think.

Even for non-dual processor aware tasks. Windows does run "smoother"
with dual processors, even for "normal" tasks.
 
Tape is incredibly unreliable - for example, if your tape drive dies
and you buy an identical replacement it's not unknown for the new
drive to be unable to read the tape.

Depends which technology.

Tape is very entrenched in the enterprise. No way would that
technology survive if it weren't reliable. Too many requirements
these days for data retention.

I'm not sure if you are in the US or not. If you are, have you heard
of Sarbanes-Oxley? (or Sarbox?) If tape couldn't perform the
required data backup for data integrity purposes, then there would be
a wholesale rush to the corporate exits for something better.
Let me ask you - have you ever performed (or tried to) a full restore
from tape?

Yes, more than once with excellent results. But I've never skimped on
the drives or media.
 
Also see "alt.comp.periphs.cdr." This group, along with what I earlier
cited, has posts that are two to five years old. There may be a group
devoted to dvd's as suitable for image archiving. It's late & I'm not
following up on that. One primary point in many posts is: Your Original
Negatives, Slides, Transparencies, and (lacking the former) Prints are
the only safeguard against time and technological change. In other
words, yes, scan & archive to digital those analog images, but for
god's sake, store those analog images you value highly as safely and
archivally as possible, for they will always be accessible to any
future technology. Of course, digital originals do not have this
guarantee.
 
It all depends on what you mean by "corruption", and by "uncontrollably".

By "uncontrollable" I mean you're just applying the noise without
consideration to the data. By contrast, interpolation will take the
data into account before coming up with an intermediate value.

I used "corruption" in the generic sense that the original data will
be modified. Of course, by that definition any editing is also
"corruption". But the difference in this case is the same as above.
With editing you're taking existing data into account, while just
applying noise randomly to an image the data is largely ignored.
That's all I wanted to demonstrate: that noise can be applied in a way
that is reversible, as long as the histogram is considered a legitimate
part of the data needed for reversing.

Why are you concerned with reversibility? One possibility is to
archive you raw scans and work on a copy. That way you can always go
back and start from the beginning. That's what I do. I always save the
original raw scans first before doing anything else.
By the way, the new http://ljl.150m.com/scans/snoise.tar.gz I've just
uploaded contains also a Windows executable.

OK, thanks, I'll get it when I send this.

Don.
 
Don said:
On Sun, 25 Sep 2005 15:36:08 +0200, "Lorenzo J. Lucchini"

[snip]

By "uncontrollable" I mean you're just applying the noise without
consideration to the data. By contrast, interpolation will take the
data into account before coming up with an intermediate value.

I used "corruption" in the generic sense that the original data will
be modified. Of course, by that definition any editing is also
"corruption". But the difference in this case is the same as above.
With editing you're taking existing data into account, while just
applying noise randomly to an image the data is largely ignored.

Well, I'm not just applying noise randomly (otherwise the operation
wouldn't be reversible anyway): I *am* taking the data into account, by
taking the histogram into account.
Why are you concerned with reversibility?

I'm not really so concerned with reversibility. In fact, I have no
problem believing that many *irreversible* transforms would show much
better results than my simplicistic algorithm.

It's just that what you said here...

--- CUT ---

[myself]
That's because this noise addition is lossless, and the "original" >
also has its histogram.

[you]
The problem is that's just physically impossible. There's only a fixed
amount of pixels in an image, and the histogram shows them all.

--- CUT --

.... is not really true.

Whether this fact has any useful practical applications, I'll leave to
more skillful eyes to judge. But it's not impossible.
One possibility is to
archive you raw scans and work on a copy. That way you can always go
back and start from the beginning. That's what I do. I always save the
original raw scans first before doing anything else.

That's certainly a reasonable thing to do. Still, if one could find a
posterization-hiding algorithm that is *both* nice-looking and
reversible (it might not exist of course), there would be no need at all
to keep a copy.

Well, as long as that's the *only* transformation applied to the
original image... so I guess this is all more theoretical than anything.
But who knows, theory sometimes have some applications one doesn't
think of off hand.

by LjL
(e-mail address removed)
 
You've not answered my basic question, which is how do you "see" noise
or posterization or other flaws with "objective" (actually
quantifiable) tools? What do noise or scanner or IR artifiacts look
like in a histogram? What is the "ruler" for scan quality and how do
you use it?
If there is a resource that explains this I am open to reading about
it. Your previous posts that I've read don't go into any detail but
just suppose the existence of such tools.
My response has been the tools I have in Photoshop 7 do not tell me
much about these flaws. This leads me to conclude that the best form
of input is therefore visual data- i.e. observation. Observation in a
controlled setting is the starting point for the "scientific method"
which then leads you to form a hypothesis and challenges to the
observation through experiment.

Don wrote:
"First of all, to do any meaningful testing you have to *disable* all
image processing!!!"
Can you explain why you would do this in a scanner driver when you will
be using the processed output for manipulation in Photoshop (why would
I do this- I don't work on gamma 1.0 raw scans in Photoshop). What are
the advantages of this and why is it more meaningful?

Don wrote:
"But *read* and *think*, don't just emotionally react! It's all there.
Explained *multiple* times!"

Once again, insulting people by claiming they are emotional,
irrational, or illogical. Thanks again, I appreciate it. Your
"explanation" could not possibly be at fault.
 
Father said:
Depends which technology.
Indeed. The DAT tapes I used to have were not good in this respect. Rumour
has it that this extends to all heliscan drives. Linear tape technology (DLT,
LTO) is far more reliable, in any case better than CDs. To get a bit back to
the OT, as good as B&W film.

-- Hans
 
Roger said:
[snip]

When I talked about having a smooth 8-bit histogram, that's for output
AFTER extensive correction with curves, selective color adjustments,
and of course sharpening in Photoshop. I have worked on files from
different sources (consumer digital cameras, flatbed scans) and found
that similar corrections gave visible problems, like posterization. VS
gives me source files with negatives that are close to what is on the
film, and that are robust enough to stand up to some real editing.

That's fine. But then you're saying that VueScan is good *because* of
objective (if a bit vague, as they've been defined in this subthread)
facts like the histogram etc.
That's the goal of the scan for me- to give me a solid starting point
that sets me up for a quality final print. I do all of the
"subjective" processing (color, grain reduction, sharpness, levels) and
making the image look the way I want in Photoshop- Vuescan is just to
get an original that looks like the negative or slide.

That's the way I see it, too.
LjL, what do you mean by scientific or objective testing and how do you
know Don has done any?

I'm not qualified to give you a decent reply on this. By scientific or
objective testing I mostly mean taking *measurements* (not visual
observations) to discover things about
- how much information (in an information-theory sense) the scanning
software loses from the "raw" scan
- how capable the scanning program is to restore important parameters of
the scanned picture (colors, sharpness, etc); I suppose this can be
measured objectively with things like test targets etc.
- how good the scanning software is to instruct the scanner so that it
extracts as much information as it is physically capable to (things like
setting exposures, setting focus, multi-sampling, possibly decent
multi-pass multi-sampling...)

Then, I don't really know what tests are good for determining what.
Actually, I mostly don't know what any tests do at all.

But I just had the impression that you were rejecting the idea of
"objective testing" itself (well, in the "scanner software" domain at
least).
I disagree wholeheartedly with such a position: even though I'm not
competent enough to know the specifics, I definitely know that the
parameters I listed above *are*, mostly, measurable. And ought to be
measured.


The answer to your second question is easy: I *don't* know that Don has
done any objective testing. He might have, or he might have not.

But please understand that I wasn't specifically defending Don or
accusing VueScan (I'd have to know both better in order to "defend" or
"accuse" either).
I was just taking a stance against what I understood to be your idea of
"testing a scanner program".


by LjL
(e-mail address removed)
 
Back
Top