The Ultimate Scanner Software vs. Photoshop Question

[Kennedy McEwen]

I also realize the problems with using the cardboard edge as a
reference. It may even skew things more than it helps because it's
slightly transparent.

Eh???

What sort of crap to Kodak peddle to you guys on the other side of the
pond? No matter what the slide mount is made of, plastic, card or
metal, I have never seen any make which transmits more light than the
densest film available. It should be, as my father used to say, "as
black as the Earl of Hell's waistcoat!".

Are you suggesting that you can see the film perforations through the
mount itself?

Your idea of loading the image back into NikonScan to do the gamma
prompted another thought! Since NikonScan can read saved PS curves an
option may be do all the editing in PS with its multitude of handy
utilities, such as Threshold (although I use my non-weighted version),
as well as inspect the image at 10 gazillion % magnification as I seem
to like to do...

Err - I didn't say NikonScan can read saved PS curves: that is something
I haven't tried and see no reason to, but it might be possible. Have
you tried that? If so, how did you do it? Nikonscan can read image
files which have been processed in Photoshop though - and any compliant
tif file, including many synthetic ones I have created myself.

BTW, don't NikonScan curves also suffer from 8-bit precision problems?
I mean, the histogram appears only 8-bit with 256 bins.

Yes and no. The Nikonscan curves are linear segments between 256 evenly
spaced points, including 0 and 65535. In this respect they are accurate
to 8 bit precision. However, and this is where Nikonscan is
significantly superior to Photoshop curves, the value at each of those
datum points is a 16-bit number, whilst Photoshop AMP files are merely
accurate to 8-bits. So, for example, in the correction that I gave you
for the shadow scan to take account of gamma, Nikonscan would not
produce the flat segments, because it has sufficient precision to define
the differentiate between all adjacent vertices.

I know. But since I seem to have a dark subject in almost every roll,
some of those deep, deep shadows are very close to unexposed...

They will always be lighter than the mount - except for the
semitransparent ones you seem to have. Even then, where the mount
sandwiches unexposed film it will still be blacker than anything on the
film.

I also noticed a green cast in the shadow image, while the nominal
scan is distinctly dirty-brown. It actually permeates the whole image,
not just the shadows. Is this color mismatch to be expected?

You should expect some sort of greenish/cyanish tint in the deep shadows
if you enhance them - that is the nature of the dyes. If you examine
the dye characteristic curve for Kodachrome, the red begins to saturate
with most density, then green and finally blue as the lowest density
dye. So a geen/cyan cast in the shadow scan is entirely expected and
merely represents what is actually on the film itself. Without the
increased exposure though, it is hard to discern in real images. If you
take some unexposed Kodachrome and look through it at a bright light
(*NOT* the sun!!) it will appear greenish too, for the same reason.

BTW, looking at the EV reduction AMP curves in Photoshop I was
surprised because I expected curves but I see straight lines. I
checked the calculations but I don't see any errors. Is that right,
should the generated curves be more like straight lines? My numbers
for 2 EV seem to correspond to your example last time:


so, I don't think I made any errors.

That is because it *is* a straight line!
That was my whole point about the amp route not having enough precision
to do the job properly. The actual gamma curve is 255 linear segments
where the ends of each segment lie on the continuous curve. Since the
correction you need for a continuous curve is just a linear gain
correction then the corrections as defined at the ends of the linear
segments are also linear. In the case of Photoshop where the end points
of each segment have only 8-bits, some of these segments are flat lines,
but that is an additional, but significant, error. My point was that
this was a fruitless exercise because you need more than 255 segments to
produce an accurate correction.

For example, say you start with data in one linear segment (which has a
certain slope) so you know exactly how to remove the gamma correction
(ignoring the 8-bit limit of amp curves) then you apply the EV
correction (dividing by 4 or 8 or whatever) to produce data that lies in
another linear segment of the gamma curve with a different slope. The
entire correction *could* be summarised as a single gain adjustment for
that data level. But the transition points where the discrete changes
in that gain occurs do not lie on the nice 256 evenly distributed points
within the 16-bit data range. I haven't worked through this, but
intuitively I would expect at least 512 points to be required, and
possibly a few more depending on the EV correction being used and how
many original gamma segments map onto the output gamma segments.

That is why I suggested that the most accurate solution available in
Photoshop would be to use a levels output adjustment - where the only
inaccuracy is the deviation of the linear gamma segments from the ideal
continuous curve. Just apply the correction that the equation produces
to a level of 255 to determine how the output should scale. This worked
pretty well with your sample images as well, although I suspect these
were 2EV shadow scans, and the result produced mean and medians which
were pretty close to the primary scan (I suspect the residual difference
was just noise, but I can't be sure from a single scan).

Finally, you noticed last time that the shadows scan was slightly
blurred. I forgot to mention that the part of the image I uploaded is
from one of the corners. So - due to film curvature - that part of the
image is indeed a little out of focus (although that should equally
apply to both scans).

OK, that probably explains it - with 4-8x less signal the defocus would
be less visible in the primary scan.

 

[Don]

Are you suggesting that you can see the film perforations through the
mount itself?

No, not the perforations, but with these difficult images I use for
testing I have to boost AG quite a bit and then the mount starts to
get quite pale.

Now, I realize it is exactly those "pale blacks" that the setting of
the black point is correcting but the cardboard edge also seems to
have a cast which appears to indicate a level of transparency.

On reflection, this cast is probably the very "mask" (CDD noise, or
whatever - you know what I mean...) I'm also correcting by setting the
black point, but there's just this appearance of transparency.

Err - I didn't say NikonScan can read saved PS curves:

I know you didn't! That's my clever, lateral thought - even if I do
say so myself... ;-)

However, your suggestion to use NikonScan to apply gamma prompted it.

that is something
I haven't tried and see no reason to, but it might be possible. Have
you tried that? If so, how did you do it?

In NikonScan invoke the Curves drop down menu (located on the right, a
downward pointing triangle). This has the options: Save Curves, Delete
Curves, etc... One of the options is:

Import Curve Settings

In the dialog box which pops up the default file type is *.ncv i.e.
NikonScan curves, but if you expand File Type, you'll also see:

Photoshop Levels and Curve (*.alv, *.acv)

I made a mental note of this when I was reading the NikonScan manual
because it struck me as a curious thing which may be useful. Yeah, I
know, it's sad - but I'm one of those weird people who actually likes
to read the manual cover to cover... ;o)

Yes and no. The Nikonscan curves are linear segments between 256 evenly
spaced points, including 0 and 65535. In this respect they are accurate
to 8 bit precision. However, and this is where Nikonscan is
significantly superior to Photoshop curves, the value at each of those
datum points is a 16-bit number, whilst Photoshop AMP files are merely
accurate to 8-bits. So, for example, in the correction that I gave you
for the shadow scan to take account of gamma, Nikonscan would not
produce the flat segments, because it has sufficient precision to define
the differentiate between all adjacent vertices.

I see! That's an important distinction, indeed.

The first time I loaded a 16-bit image in PS I was confused that the
histogram display was still only 256 distinct bins and had an uneasy
feeling about it.

You should expect some sort of greenish/cyanish tint in the deep shadows
if you enhance them - that is the nature of the dyes. If you examine
the dye characteristic curve for Kodachrome, the red begins to saturate
with most density, then green and finally blue as the lowest density
dye. So a geen/cyan cast in the shadow scan is entirely expected and
merely represents what is actually on the film itself. Without the
increased exposure though, it is hard to discern in real images. If you
take some unexposed Kodachrome and look through it at a bright light
(*NOT* the sun!!) it will appear greenish too, for the same reason.

Yes, since I wrote that, I was actually thinking about those dye
characteristic curves for Kodachromes (the link you posted earlier)
and wondered if that was the reason.

BTW, my method of adjusting this color mismatch is by using Means. I
realize that this method may have inaccuracies but the histogram band
I'm correcting is quite narrow so I think any inaccuracies are
probably fairly minor. Besides by fine tuning I could probably even
take care of that as explained below.

For example, take the two images I posted and first boost them both
by, say, applying a Curve with RGB Master Input=88, Output=168 so you
can see what is going on. After that do a Histogram on both and note
the Means. I get:

for 0.0 ev
R: 48.78
G: 42.46
B: 44.94

for 3.0 ev (actually 2.0 as we've discovered...)
R: 64.12
G: 90.09
B: 101.77

Using those values (rounded) I create a curve with the following
settings:
R: Input=64, Output=49
G: Input=90, Output=42
B: Input=102, Output=45

Applying that curve to the shadows scan (instead of EV adjustment i.e.
AMP curves from your formula) creates an amazing color match!

My question is (out of curiosity and just trying to understand) what
is it in terms of theory I'm doing here?

I mean, it does work and seems to kill two birds with one stone;
adjust brightness *and* correct color at the same time!

As I mentioned earlier I can fine tune this further by creating Curve
points for *every* histogram bin (in the desired range) to achieve,
what to me appears as, perfect color correction with no inaccuracies.
But I'd like to hear your thoughts.

That is because it *is* a straight line!

OK... Phew... I know that EV adjustment is linear, but I just thought
that inclusion of gamma correction would "curve" the result somehow.

On reflection, that's silly because removing and applying gamma
cancels any effects... Never mind...

Don.

 

[Kennedy McEwen]

No, not the perforations, but with these difficult images I use for
testing I have to boost AG quite a bit and then the mount starts to
get quite pale.

Now, I realize it is exactly those "pale blacks" that the setting of
the black point is correcting but the cardboard edge also seems to
have a cast which appears to indicate a level of transparency.

And since the cardboard mount should be very black indeed this indicates
that your black point is seriously off. The colour cast is just
indicative of the CCD's colour response and the different exposure of
each LED for the default, giving a tint to the black produced by all of
the scattered light in the scanner.

On reflection, this cast is probably the very "mask" (CDD noise, or
whatever - you know what I mean...) I'm also correcting by setting the
black point, but there's just this appearance of transparency.

Yes, poor blacks make things look lighter and translucent. ;-)

In NikonScan invoke the Curves drop down menu (located on the right, a
downward pointing triangle). This has the options: Save Curves, Delete
Curves, etc... One of the options is:

Import Curve Settings

In the dialog box which pops up the default file type is *.ncv i.e.
NikonScan curves, but if you expand File Type, you'll also see:

Photoshop Levels and Curve (*.alv, *.acv)

Hadn't noticed that, but alv and acv files are just 13-point curves, if
I recall correctly, so even less precise than the amp definitions you
have been using. I don't know what the ncv format is, but it produces a
723byte file, which doesn't seem to be enough precision unless it is
merely defining input and output black and white levels and gamma for
all three colours, with Nikonscan then applying the appropriate curve
fit. Again, it doesn't look like there is enough data there to do what
you want.

The first time I loaded a 16-bit image in PS I was confused that the
histogram display was still only 256 distinct bins and had an uneasy
feeling about it.

I guess they have tried to keep some commonality of descriptions, such
as maintaining the 8-bit "level" format and such like. From an artistic
user's perspective that would be important. When trying to implement an
accurate function it isn't.

BTW, my method of adjusting this color mismatch is by using Means.

From a technical perspective, you should be using the median, since the
mean will be skewed by the asymmetric distribution. Median is a closer
approximation of the peak in an asymmetric histogram.

My question is (out of curiosity and just trying to understand) what
is it in terms of theory I'm doing here?

What you are doing is committing one of the cardinal sins of colour
manipulation - trying to correct a black point error by adjusting the
gamma and/or contrast. You can get this to work, but only accurately at
one level. The further you go from that, the worse the error gets until
eventually it is worse than the blackpoint error you started with.

Take a simple linear example. You have a black point error of 10, say.
That means that what should be a level of 25 is actually 35. You can
adjust this using a contrast (gain = 25/35) so that the end result is
25. However, for levels which should be, say, 45 and appear as 55
because of the black point error, that erroneous contrast adjustment
would produce an output of 39, which is not what it should be. This is
fairly trivial and obvious in a monochrome linear system, but just as
relevant in a colour, non-linear system.

I mean, it does work and seems to kill two birds with one stone;
adjust brightness *and* correct color at the same time!

Only over a limited range. Whether that range is enough in this
instance is something that could be determined objectively, but is
probably easier to tell subjectively by viewing a range of different
corrections. You run the risk that one or two might not work as well as
others though.

OK... Phew... I know that EV adjustment is linear, but I just thought
that inclusion of gamma correction would "curve" the result somehow.

On reflection, that's silly because removing and applying gamma
cancels any effects... Never mind...

Yes, but it only cancels out when the gamma is applied as a continuous
curve of the form Y = X^g. It isn't as straightforward as that when the
gamma correction is actually made up of linear segments. You need a lot
more segments to take account of the mapping from gamma-linear and
linear-gamma after the gain adjustment. Since you don't have such an
option then the continuous curve approximation is the best you are
likely to get.

 

[Don]

And since the cardboard mount should be very black indeed this indicates
that your black point is seriously off.

I'm aware of that and it was actually intentional because I was
planning to adjust later in PS. By boosting AG to such extremes (~3
EV) it's only to be expected that the whole histogram would move
radically to the right making blacks "anemic".

The colour cast is just
indicative of the CCD's colour response and the different exposure of
each LED for the default, giving a tint to the black produced by all of
the scattered light in the scanner.

I thought that that AE does not mess with individual channel AG
(individual LEDs) and only modifies the Master.

Long story, but in order to extend the amount of AG beyond the
hardcoded NS limit of 4 (Master and individual combined) I ran some
tests back on the LS-30 (using NS 3) to try and determine the amount
of boost AE set for a particular slide. In that particular case, there
was a +2.1 deviation from the nominal, manual scan with AG=0. Even
though the slide had a serious mismatch between individual channels
(the Kodachrome thing) only the Master AG had been modified by AE.

In other words, the manual scan with AG=+2.1 and the automatic (AE)
scan with AG=0 were identical i.e. identically exposed. Both had the
same amount of cast, which lead to me conclude that AE only modifies
Master AG.

Hadn't noticed that, but alv and acv files are just 13-point curves, if
I recall correctly, so even less precise than the amp definitions you
have been using.

I don't know about Levels but I guess they would be only 3-point
settings (white, gray and black). The *.acv files i.e. Curves can have
up to 16 points. Still, quite far from AMP curves.

I don't know what the ncv format is, but it produces a
723byte file, which doesn't seem to be enough precision unless it is
merely defining input and output black and white levels and gamma for
all three colours, with Nikonscan then applying the appropriate curve
fit. Again, it doesn't look like there is enough data there to do what
you want.

Well, since both *.ncv and *.acv seem to only have 8-bit precision
with 256 bins I don't think there is that much difference.

Anyway, it was just a lateral thought of trying to combine the best of
both worlds. Actually, more of a thought experiment.

I guess they have tried to keep some commonality of descriptions, such
as maintaining the 8-bit "level" format and such like. From an artistic
user's perspective that would be important. When trying to implement an
accurate function it isn't.

That's the key! PS is really a tool for artists. For example, just
look at the various blending modes. I would've expected a mathematical
description e.g. AND, OR, NOT and so on, so you know what is being
done. Instead, PS uses "flowery" language such as "Color dodge" which
is totally meaningless to me.

I was given a link to the definition of some of them in the PS
newsgroup, but I would much prefer a definitive statement from Adobe.
I ended up running tests and tabulating the results in Excel... until
I got sidetracked by other things... ;-)

From a technical perspective, you should be using the median, since the
mean will be skewed by the asymmetric distribution. Median is a closer
approximation of the peak in an asymmetric histogram.

Actually, I started with median, but for these narrow bands it
appeared (subjectively) that mean produced better results.

I suppose, ideally, I should really only use 1 bin per adjustment
(instead of 32 as I currently do, i.e. 0-31) and then define one point
per bin in *.amp files using medians.

But, that's where I have to step back, do a reality check and ask
myself if going to so much effort is really worth it...

What you are doing is committing one of the cardinal sins of colour
manipulation - trying to correct a black point error by adjusting the
gamma and/or contrast. You can get this to work, but only accurately at
one level. The further you go from that, the worse the error gets until
eventually it is worse than the blackpoint error you started with.

That confirms what I got in my early attempts (LS-30 days) where I
would use a single point for the whole image (my "meta pixel"). That
didn't work, of course, for any image with a wide dynamic range. It
did work for the proverbial picture of a black cat in the dark,
though... ;-)

Another reason is that if a channel is clipped in the highlights then
no amount of adjustment can bring that particular pixel down to the
nominal scan level because of missing (clipped) data. Of course, I no
longer try to adjust the whole image but only the shadows where such
clipping does not occur.

Take a simple linear example. You have a black point error of 10, say.
That means that what should be a level of 25 is actually 35. You can
adjust this using a contrast (gain = 25/35) so that the end result is
25. However, for levels which should be, say, 45 and appear as 55
because of the black point error, that erroneous contrast adjustment
would produce an output of 39, which is not what it should be. This is
fairly trivial and obvious in a monochrome linear system, but just as
relevant in a colour, non-linear system.

Yes, that's what I noticed empirically at which point I realized I
need to define more points i.e., limit the range of the adjustment.

Don.

 

[Kennedy McEwen]

I don't know about Levels but I guess they would be only 3-point
settings (white, gray and black).

You might want to look into this. What you refer to as "Grey" is
actually gamma and results in 255 linear segments with 16-bit precision
on the vertices when applied in Nikonscan.

 

[Don]

You might want to look into this. What you refer to as "Grey" is
actually gamma and results in 255 linear segments with 16-bit precision
on the vertices when applied in Nikonscan.

I realize that, but it was my impression that only the new gamma value
was noted, or more accurately the amount to be applied. What I mean by
this is when you change gamma from, say, 2.2 to 1.0 you only need to
store the 0.454545... and calculate the rest on the fly.

Anyway, it's not really important. Using PS settings from NS was just
a lateral though...

Don.