Nikon Color Management and Profile Quality

[Don]

Don, is your program usable by other people? I'd be interested in
using it.

It's just something I wrote for myself. But the theory behind is known
(I explained it in a long message recently) and it's not that hard to
program.

The major difference from the "Twin Scan" message is that I now
average *every* histogram bin (not just the bin where the two images
join) and create a 16-bit lookup table out of that. I then simply
apply this lookup table to the shadows scan. (Just for fun, I also
converted this 16-bit lookup table into a Photoshop AMP curve as a
roundabout way to plot the data.)

While I'm at it, I then also note the point at which pixels in the
shadow scan become clipped. But I've found this to be quite far
removed from the "join" value so it's not an issue (but I wanted to
make sure). For example, I currently use bin 36 - 40 (on the 8-bit
scale) for the join, while clipping usually starts (at least in my
test images so far) way over 128.

Don.

 

[Don]

I've got better things to do with my time
than disassembling Windoze apps. I guess
you really are a glutton for punishment.

Predictably, in the absence of facts - and having painted yourself in
a corner - you've now resorted to insults. :-(

I'm sorry, but you've been around here long enough to know I'm not
interested in pointless name calling.

Don.

 

[Don]

It is, however, interesting to note that both Don and yourself are
slating NikonScan based on experiences with the LS-30 which, of course,
uses completely different profiles and files from any of the other
scanners where the users are perfectly satisfied with the results.

After all the shouting it's refreshing to see Occam's razor at work in
this group!

That is indeed the crux of the matter!! It's also why - at least this
time - I carefully indicated that my experiences were limited to the
LS-30.

It
would be useful to determine some hard evidence of whether the problem
is the intrinsic limitation of this older scanner or an actual error in
the specific profile itself.

My LS-30 is in its box right now, and - even if I had the time - it
would be a major production for me to connect this SCSI scanner to my
notebook right now.

However, the images I originally posted years ago are still there, but
I don't know how useful they would be in this context...

Don.

 

[Jean]

....snip


<cheap_shot>
It's (at least) two generations newer than the current Mac OS. (Unix was
1970s, VAX VMS was 1980s, and NT was 1990s.)
</cheap_shot>

Cheap shots aside<g>, all those OSes are stable _by design_. Neither the old
Mac OS nor Win 95/98/ME have a snowball's chance in hell of not crashing,
given their design.

Friends don't let friends use Win 95/98/ME.

David J. Littleboy
Tokyo, Japan

Windows ME has more in common with XP than it does with 95 and 98. It
has been exceptionally stable for me and has never crashed in 5 years.

Jean

 

[Philip Homburg]

Cheap shots aside<g>, all those OSes are stable _by design_. Neither the old
Mac OS nor Win 95/98/ME have a snowball's chance in hell of not crashing,
given their design.

From a practical point of view this is stupid. There have been plenty of
netscape versions that were quite buggy. Now, the OS may not crash in
that case but you can still lose quite a bit of work.

On the other hand, my Win98SE system not crash. I don't know why it doesn't
crash, that is just the way it is.

I prefer a systems approach to reliability: if one component is not up to
its task, the whole system is broken. Plenty of Unix system come with
(or at least, used to come with) GUI apps that crash.

Other other hand, with selected applications both Apple OS and Win9x can
be stable.

I consider all Microsoft operating systems complete and utter junk. So I
see absolutely no point in upgrading system that does not crash, to one
that is broken in a completely different way (even when Windows does
not crash, it is still broken).

 

[John]

Well, to add to the confusion in that case, I use the same scanner. I
always scan my images with NCM on, unless I am conducting measurements
and need access to the raw data when I switch it off and set gamma to
unity. However I have never encountered any of the posterisation or
other artefacts that have been discussed in this thread.

The plot thickens! I may have to revisit this when I have more time (not in
the near future) to get a more quantitative feel, but I will try to recall
my problems as I remember them.

I, like you, used to always scan with NCM on and for the most part, it
seemed to do a good job, Colour balance was always close (Kodachromes
excepted). In fact, when I first read Bruce Fraser's review of the 4000 and
his comments re. NCM's shadow clipping, I didn't believe it. However, once
aware of the possibility, I began to notice there was such a tendancy, but
not so much as to be a problem for me.

The crunch came when a customer of mine brought in some slides which he
wanted prints from - these slides turned out to be those ultimate beasts
from Hell - underexposed Kodachromes. I coped with some of them, but the
worst just produced black shadows. Scanning with NCM off straight into
working space and hand tweaking them sorted the problem. Don't get me
wrong - they were by no means brilliant, but I got something poor instead of
something unusable. I have since had other similar experiences.

The other approach I have tried is to scan with NCM off and then assign
Nikon's canned profile to it and convert to Adobe RGB (1998) or whatever. I
used this approach on some of my own shots taken in Kintyre a couple of
years ago and the results were excellent. (Mind you, with such fine views
they could hardly be otherwise, but I digress!) I thought I had cracked it -
this method means I can scan once and decide on an individuad basis whether
to use the profile or hand crank. However, the limitations of the profile
showed up with another job where I noticed that some distant woodland on a
photo had lost all its detail and just became a flat green. Again, hand
tweaking from the raw data produced no such problems.

To be honest, I can't remember if I have experienced posterisation with NCM
itself or whether it was just when using the canned profile in Photoshop.
Shadow clipping has definitely occured with both.

Of course, it may be true to say (in my case) that Nikon CMS gives good
results maybe 80% of the time, if the image is not that demanding or if the
clipping plain and simply doesn't matter. Given the time that it saves when
it does work, it may be bad advice to suggest to anyone that they should
always turn it off. That decision has to be based on workflow issues and
starting material. However, my aim in this thread was not to give advice
(and I didn't), but to recount my experiences with NCM when it had caused
problems. This was in direct response to the OP who wanted his 'memory
jogging'. In addition, Rafe's statement that NCM simply tags the raw data
with a working space profile could have led the reader onto a very rocky
shore, so needed to be corrected. Clearly there is a range of experiences
yet to be rationalised - some people have problems, others don't. Perhaps
there is another factor which we are not considering - with calm discussion,
we may eventually get there. At this stage, the wisest course of action is
for every user to be aware of where to look if problems are to arise.

One thing which does confuse me, and you may be able to help here, is how
NCM manages to use an LUT profile even though auto-exposure is enabled.
According to others who post here and perceived wisdom, this is a complete
no-no as you need to keep a consistent set of scanning conditions for a
given profile to be valid. This is also apparently why Ed Hamrick uses
matrix profiles in VueScan.

By the way, do you scan with autoexposure on or off? All my scans have been
with it on, but I haven't worried on the basis that there was no clipping
with NCM turned off and therefore no reason to suspect it as the culprit. Of
course, it is possible that the exposure regime is different with NCM on,
although I can't think why it should be and it certainly isn't documented.

 

[Anoni Moose]

<cheap_shot>
It's (at least) two generations newer than the current Mac OS. (Unix was
1970s, VAX VMS was 1980s, and NT was 1990s.)
</cheap_shot>

Windows NT (XP is a renamed newer version) was architected by the
head architect honcho who worked on VAX VMS that Microsoft
"stole" from DEC. Of course, NT originally didn't have
a GUI and it wasn't designed for Intel processors (designed
for RISC processors like MIPS and such). Those things were
added in late ("moving targets" are so wonderful).

Anyway, your cheap shot brings up XP's heritage from VMS.

Mike

 

[Kennedy McEwen]

One thing which does confuse me, and you may be able to help here, is how
NCM manages to use an LUT profile even though auto-exposure is enabled.
According to others who post here and perceived wisdom, this is a complete
no-no as you need to keep a consistent set of scanning conditions for a
given profile to be valid.

Provided the relative exposure of each colour remains in the same as a
consequence of the autoexposure, then the profile will remain perfectly
valid. For example, if the autoexposure determines that the exposure
needs to be increased by 53% to adequately fill the CCD dynamic range
then, provided that the 53% is applied equally to the red, green and
blue exposures, the colour response, and hence the required
transformation profile, should remain the same.

It is only when the relative exposure of the individual channels change
that the character of the profile changes. Even then it should be
relatively straight forward to determine the exact effect of the change
and adjust either the data or the profile accordingly. I haven't gone
into this in great detail, but I intuitively feel that the highly
monochrome nature of the Nikon illumination system should simplify this
compared to the broad spectral response of a conventional scanner.

By the way, do you scan with autoexposure on or off?

Usually. The autoexposure simply sets the brightest part of the image
to be just below the saturation level of the CCD, thus optimally
utilising the limited dynamic range available.

 

[rafe bustin]

In addition, Rafe's statement that NCM simply tags the raw data
with a working space profile could have led the reader onto a very rocky
shore, so needed to be corrected. Clearly there is a range of experiences
yet to be rationalised - some people have problems, others don't. Perhaps
there is another factor which we are not considering - with calm discussion,
we may eventually get there. At this stage, the wisest course of action is
for every user to be aware of where to look if problems are to arise.

My statement was based on the fact that I'd worked
with color management turned off for a couple of
years (in NikonScan) and then with CM turned on --
and nothing much changed.

There may be other factors at play. In particular,
90% of what I scan are color negatives, not slides.
I use Dane Kosaka's "inversion" procedure, whereby
negatives are scanned as positives and inverted with
NikonScan's curves tool. Color management is on,
and set to AdobeRGB. (My monitor is hardware profiled
and calibrated, and I print through printer profiles
that I make myself with Profile Prism.)

That said, I simply haven't seen the posterization
that you and Don speak of.

When I scan, I set black and white points individually
for each color channel of each frame, and gamma tweaks
where necessary (per channel and/or composite RGB.)

I use radically different analog gains for the three
channels, typically something like

Master: +0.5
Red: -0.3
Green: +0.7
Blue: +1.7

Again: this is for scanning C41 as slides.

There's a school of thought in scanning that
goes for color-managed output, and another
that goes for "optimum information capture."
I subscribe to the latter.



rafe b.
http://www.terrapinphoto.com

 

[Philip Homburg]

I use radically different analog gains for the three
channels, typically something like

Master: +0.5
Red: -0.3
Green: +0.7
Blue: +1.7

Again: this is for scanning C41 as slides.

I do the same thing, except that I reverse in Photoshop. I have similar
analog gain settings.

I doubt that scanning negative is a good way to evaluate color management in
NS. If, for example, the high-lights in the scan get clipped, then, because
it is a negative, in the final image the shadow areas will be clipped, but
that will probably go by unnoticed.

There's a school of thought in scanning that
goes for color-managed output, and another
that goes for "optimum information capture."
I subscribe to the latter.

Those two can be combined, except that actually doing so may get a bit
tedious. I don't know if analog gain can be modeled properly. If it can
be modeled, color management should not require additional work. Otherwise
you would have to profile the scanner at each analog gain setting you
use

 

[John]

On Mon, 23 May 2005 15:48:52 +0100, "John"


My statement was based on the fact that I'd worked
with color management turned off for a couple of
years (in NikonScan) and then with CM turned on --
and nothing much changed.

There may be other factors at play. In particular,
90% of what I scan are color negatives, not slides.
I use Dane Kosaka's "inversion" procedure, whereby
negatives are scanned as positives and inverted with
NikonScan's curves tool. Color management is on,
and set to AdobeRGB. (My monitor is hardware profiled
and calibrated, and I print through printer profiles
that I make myself with Profile Prism.)

That said, I simply haven't seen the posterization
that you and Don speak of.

I may be guilty of loose terminology here, so just to clarify, what I call
'posterisation' occurs outside the shadows, for example, my disappearing
tree detail was actually in the midtones. However, to be honest, I can't
remember if I have ever seen this occuring with NCM - it happened when I
used the Nikon 'canned' profile and converted to Adobe RGB (1998) in
Photoshop. What I have seen in NCM is clogging of the shadows, which is what
I refer to as shadow clipping. If this is the case, it may account for the
differences in what you and I have seen, since slides are more demanding
contrast wise. In addition, your top-end Coolscan 8000 may perform better in
dense areas - am I right in thinking that it has a higher Dmax then the
4000?

Definitely a case of 'your mileage may vary!'

When I scan, I set black and white points individually
for each color channel of each frame, and gamma tweaks
where necessary (per channel and/or composite RGB.)

I use radically different analog gains for the three
channels, typically something like

Master: +0.5
Red: -0.3
Green: +0.7
Blue: +1.7

Again: this is for scanning C41 as slides.

There's a school of thought in scanning that
goes for color-managed output, and another
that goes for "optimum information capture."
I subscribe to the latter.

Ah yes, but I am greedy - I want the best of both worlds I don't think
that theory precludes it, although I don't expect it to be easy. However,
the rules change drastically for negatives - the benefit of capture colour
management is much less clear in this case. To be honest, I use VueScan
almost exclusively for negs these days. I agree that the method you outline
above will give the best possible results. The problem I have is that much
of the scanning I do is for customers, so unlike yourself, I have no control
over the starting material and I didn't take the original pictures. The
challenge for me is just to try to eliminate as much of the guesswork as
possible.

 

[John]

Provided the relative exposure of each colour remains in the same as a
consequence of the autoexposure, then the profile will remain perfectly
valid. For example, if the autoexposure determines that the exposure
needs to be increased by 53% to adequately fill the CCD dynamic range
then, provided that the 53% is applied equally to the red, green and
blue exposures, the colour response, and hence the required
transformation profile, should remain the same.

It is only when the relative exposure of the individual channels change
that the character of the profile changes. Even then it should be
relatively straight forward to determine the exact effect of the change
and adjust either the data or the profile accordingly. I haven't gone
into this in great detail, but I intuitively feel that the highly
monochrome nature of the Nikon illumination system should simplify this
compared to the broad spectral response of a conventional scanner.

Usually. The autoexposure simply sets the brightest part of the image
to be just below the saturation level of the CCD, thus optimally
utilising the limited dynamic range available.

So if I understand you correctly, I should be able to use a custom profile
(say one made with LCMS) and use it for different exposure levels (or indeed
auto-exposure) as long as I don't alter the relative colour balance with
analogue gain? I suppose the simplest way would be to try it!

 

[Bart van der Wolf]

Provided the relative exposure of each colour remains in the same as
a consequence of the autoexposure, then the profile will remain
perfectly valid.

This statement can be interpreted in two ways, one is correct, the
other may be incorrect.
If you mean that if the exposure level (near CCD saturation for the
most transparent film area) is similar to the exposure level of the
calibration, then I'd agree.
However, if you mean (and I'm afraid that that might be the case) that
as long as the relative R, G and B exposures don't change the color
stays the same regardless the level, that will not be true in all
cases.

One reason is that for some colors, a channel may clip beyond a
certain luminance level (either at 0 or at 255 in 8-bit/channel mode).
To make matters worse, most given colorspaces are not all that
perceptually uniform in all dimensions as can be checked with this CIE
color calculator:
<http://www.brucelindbloom.com/index.html?ColorCalculator.html>

Try checking the "Scale RGB" box and fill in 128 128 128, now click
the RGB button. Now at the top, change the Y (is the luminocity
component) coordinate from 0.215860 to 0.2 (use a decimal point or
comma in line with your regional settings) and click the XYZ button.
You will see that the RGB equivalent is no longer "neutral" despite
the fact that only luminocity was decreased, in this case by 7%.

Bart

 

[Kennedy McEwen]

This statement can be interpreted in two ways, one is correct, the
other may be incorrect.
If you mean that if the exposure level (near CCD saturation for the
most transparent film area) is similar to the exposure level of the
calibration, then I'd agree.
However, if you mean (and I'm afraid that that might be the case) that
as long as the relative R, G and B exposures don't change the color
stays the same regardless the level, that will not be true in all cases.

Neither of these "interpretations" is what I meant, Bart, so perhaps I
should try to elaborate.

Autoexposure adjusts the exposure of the CCD to the image so that the
most transparent part of the image always produces approximately the
same peak(r,g,b) value without risk of saturating the device (ie. with
sufficient safety margin such that noise in the actual exposure will not
cause subsequent saturation). Hence the issue of saturation does not
arise, as below, whether autoexposure is utilised or not.

One reason is that for some colors, a channel may clip beyond a certain
luminance level (either at 0 or at 255 in 8-bit/channel mode).

Similarly, the issue of conversion of data converting between
colourspaces differently depending on the level of r,g,b is also
irrelevent because the function of the autoexposure is to ensure that
images of different density produce the same r,g,b levels.

To make matters worse, most given colorspaces are not all that
perceptually uniform in all dimensions as can be checked with this CIE
color calculator:
<http://www.brucelindbloom.com/index.html?ColorCalculator.html>

Try checking the "Scale RGB" box and fill in 128 128 128, now click the
RGB button. Now at the top, change the Y (is the luminocity component)
coordinate from 0.215860 to 0.2 (use a decimal point or comma in line
with your regional settings) and click the XYZ button. You will see
that the RGB equivalent is no longer "neutral" despite the fact that
only luminocity was decreased, in this case by 7%.

However, as you should note, this example compares the transform of
different data - not the same data achieved by different exposure . If
a different exposure is required to achieve the same data levels from
different film densities, the resultant data (being identical) will
transform in exactly the same manner no matter what target colourspace
you choose. In other words, the autoexposure is simply achieving
normalised data levels in the scanned image and the resulting data will
still represent the same colour space gamut and hence require the same
profile correction to achieve a standard colour space.