"Raw" file issues?

Jeremy Nixon wrote:

Once again, nitwit: I have data that hints that Canon has a special
black-level estimator for long exposure images.

Good for you.

I know, it is. What data do you have, homeboy?

This estimator, if it exists, would be built into Canon CR2 file
decoders.

None of this is known to DNG. (I am still awaiting a DNG spec
citation.)

Then I'm awaiting an explanation of why DNG should have to know about it.

Ohhhhh, I dunno, in order to make a better black level estimate?

Given that it *already works* in DNG *right now*, I think it's pretty
clear that it doesn't, any more than it has to know about every other
camera that comes out.

Well, maybe your code is making non-optimal estimates. Oh my. Oh me.

I'm currently processing my camera's RAW files after converting to DNG.
My camera didn't exist at the time the version of Camera Raw I'm using
was written. Yet, somehow, it works perfectly. How do you suppose that
is? How could DNG possibly "know about" my camera's unique RAW format?

Because it doesn't have to, of course.

Your converter can make as many assumptions as it likes, of course.
Maybe you like guessing: I like knowing.

Given this context, why should I use DNG over CR2?

You shouldn't. No one is trying to make you. But neither should you
argue against DNG based on faulty understanding of what it is.

Take your straw-men and shove 'em. Now answer the question as posed:
if Canon software can render a better image than a DNG decoder, why use
the DNG decoder?

You
are assuming that something Canon is doing would require special
consideration in the RAW file format; it does not require any such
thing.

I am not assuming Canon is doing something: _I KNOW THEY ARE_.
Remember the "data" that is "good for me"?

If something Canon is doing results in their RAW conversion software
doing a better job than third-party software does, that is a completely
different thing and it has nothing, at all, to do with the RAW file format.

Canon's converters will do a better job becaues they have knowledge of
(a) their hardware, but more importantly, (b) specific information in
their CR2 files which directs superior handling of the raw image data.
SHOCK OF SHOCKS: some of this information may not be encodable in the
DNG file!

wrote:

Then I'm awaiting an explanation of why DNG should have to know about it.

Ohhhhh, I dunno, in order to make a better black level estimate?

DNG does not make a black level estimate at all, so making a better one
is really not relevant.

The information you're using to make that black level estimate can be
preserved in a DNG file, though, so that software that knows how to use
it will be able to.

Take your straw-men and shove 'em. Now answer the question as posed:
if Canon software can render a better image than a DNG decoder, why use
the DNG decoder?

Absolutely no reason at all. None whatsoever. No one is trying to say
that you should do that. Indeed, if you prefer the results you get with
Canon's software, I would say that you should use it.

I am not assuming Canon is doing something: _I KNOW THEY ARE_.
Remember the "data" that is "good for me"?

You know Canon is doing something; you assume that it will require
specific support in the RAW file format, and that assumption is not
correct.

Canon's converters will do a better job becaues they have knowledge of
(a) their hardware, but more importantly, (b) specific information in
their CR2 files which directs superior handling of the raw image data.

If Canon's converters do a better job, then go ahead and use them.

SHOCK OF SHOCKS: some of this information may not be encodable in the
DNG file!

Except that it is, so that doesn't apply. It *is* possible to do something
that can't be encoded by the existing DNG specification, but Canon hasn't
done so, nor has any camera since (I believe) the Fuji 2-level sensor
thing.

--
Jeremy |

Barry Pearson wrote:

The camera has some existing sensor configuration, such as Bayer, or
Fujifilm SR, or Sony 4-color,

Indeed, I don't think the Sony 4-color requires any special consideration
in DNG, as per some statements I've seen from Thomas Knoll. It's still a
Bayer pattern, and the particular colors in use apparently don't matter
since they are specified in the DNG file.

I'm still hazy about some of the details. But what I've said above is
consistent with everything I've learned and tested so far. I have
converted D2X and 350D Raw files into DNG, and processed them in ACR
2.4 under CS. CS Browser shows thumbnails for the DNG files just as it
would any Raw files that it recognised, and ACR 2.4 can process them.
But it just shows place-holders for the original Raw files, and refuses
to open the latter because it doesn't recognise the format.

I've used it for my D2x files, too, in ACR 2.4, with complete success.
It still does a better job than either dcraw or Nikon's own software,
despite having been written before the D2x existed.

--
Jeremy |

In message . com,
" wrote:

Jeremy Nixon babbles:

The native RAW files don't have to "handle" it, either.

Once again, nitwit: I have data that hints that Canon has a special
black-level estimator for long exposure images.

This estimator, if it exists, would be built into Canon CR2 file
decoders.

None of this is known to DNG. (I am still awaiting a DNG spec
citation.)

Thus no current DNG decoder uses this special estimator.

It is possible that the DNG ENcoder takes it into account, though.

That's the way DNG 3.1 deals with 20D banding, I think; when it writes
the DNG, it alters the RAW data so that it needs the same blackpoint for
every line.

Given this context, why should I use DNG over CR2?

Oh, you might argue, but Canon could make a DNG decoder that knows all
about that. Shall I respond to this now, or can you figure out the
problem on your own?

Or, Canon might allow for the camera to record this information, but
Canon never actually uses it; just like they don't use the individual
lines' black data in their RAW processors, but DNG 3.1 *does*.
--


John P Sheehy

In message ,
Jeremy Nixon wrote:

Then I'm awaiting an explanation of why DNG should have to know about it.
Given that it *already works* in DNG *right now*,

What do you mean by "already works"? How do you know the conversion
couldn't be better? DNG does not keep all black pixels; it has a tag
that *may* report the difference for each horizontal and vertical black
line from the global blackpoint it writes to the file.
--


John P Sheehy

In message ,
"Ryadia@Home" wrote:

Olympus (the digicam my wife has) has a TIFF capture for their idea of
RAW data. Now this is an image file and it can be edited direct from the
camera and saved straight back to the same file or posted unaltered for
viewing with your "standard" viewers.

One notch above JPEG, in other words ... no extra highlight level
resolution, no extra highlights beyond what becomes 255 in the output
file, etc.
--


John P Sheehy

wrote:
Jeremy Nixon wrote:
[snip]
Except that this is not correct. DNG does *not* need specific support
for all camera features in order to preserve them; indeed, that's part
of the whole point of the thing.

Even taken alone, your statement is nonsensical. It is isomorphic to
"You don't need an arm in order to use an arm."
[snip]

Let's step back to see what is being talked of he

DNG is a specification of a file format. It can cater for a whole range
of sensor configurations, including Bayer-like, Foveon-like,
Fujifilm-like, Sony-like, and others. It has various parameters that
provide extra details for each of these.

Any particular DNG image file uses just the subset which this camera
needs, out of the vast range of possibilities. So that file is doing 2
important things, as far as this discussion is concerned. It identifies
those details of the camera that will be needed when processing the
image. And it supplies the image data itself.

It isn't important whether the DNG specification took this particular
camera model into account. What matters is that the specification has
sufficient richness to cater for this camera model. So a new camera
model that differs only in bits-per-pixel, or number of X and Y pixels,
or different look-up tables to transform the sensor data according to
colour temperature, can be catered for.

It appears to be the case that DNG specification version 1.1.0.0,
released in January, had sufficient richness for the 350D and D2X
details. In fact, probably version 1.0.0.0, released last September,
already had sufficient richness.

It is beyond doubt that ACR 2.4, released in January, already had the
code within it to read DNG files generated from D2X NEFs and 350D CR2s,
and process those images. I've tried this route, and I guess that by
now lots of other people have. I wonder whether ACR 2.3, released in
September, also had sufficient code within it to handle those DNG
files? I suspect it did, because there are only small differences
between versions 1.0.0.0 and 1.1.0.0.

Yet neither ACR 2.3 nor ACR 2.4 can read a D2X NEF file or a 350D CR2
file. It takes specific knowledge of a camera model to process Raw
files, and ACR 2.x didn't have that knowledge. (Neither, in fact, did
ACR 3.0).

This shows the power of DNG. You can write code to process files
conforming to particular versions of DNG, and you can cater
automatically for cameras that you haven't heard of, and that may not
have existed when you wrote the code. The advantage for the camera
manufacturer would be that their new camera is immediately supported by
all Raw processors that catered properly for the chosen version of DNG.
If a future D3X or 375D output in DNG version 1.1.0.0, then ACR 2.4
could process the images.

When new technology cannot be catered for by the current DNG version, a
new version, for example 2.0.0.0, will be needed. That presumably can't
be processed by ACR 2.4, nor other Raw processors. Upgrades will be
needed.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

wrote:
Barry Pearson wrote:

Frankly, I haven't a clue whether the "unique black level estimation
for long exposure images" is catered for by the current version of DNG.

It isn't.

What evidence is there for this?

[snip]
Nikon (or whoever) has likely has something that is being dropped on
the floor when you convert to DNG. This is the basic point (that is
apparently lost on too many people): without specific support for all
features present in a camera, DNG is necessarily lossy.

I've just covered this in another response. What matters is whether DNG
has sufficient richness for all the details of the camera model needed
for processing the Raw file. It appears to me that it has for the D2X
and the 350D, because I could get good images via this route. But I
accept that this was just for a limited number of images. I didn't
process any that had a long exposure.

Had I tried such images, what problems would I have had? Is it
something that couldn't be handled even in theory by Raw processors
that didn't read the "unique black level estimation for long exposure
images"? Or simply something that has to be catered for another way?

Remember the fuss about D2X WB encryption. DNG files generated from D2X
Raw files don't (currently) properly identify the "as shot" WB, so in
that sense those *files* are lossy. And for a lot of photographers,
this turns out to be unimportant. But it doesn't mean that the DNG
*format* is inherently lossy; what is lossy is the conversion process
when performed by software that can't get at the WB. If the D2X camera
were to output DNG files, it might well choose to do the job properly,
and avoid loss. An open source DNG Converter, unconstrained by Adobe's
fears of legal action, might write more complete DNG files from the
D2X. I think we will see some interesting open source, and other
independent, applications of DNG over the next year or so.

[snip]
The existence of "PrivateData" is the problem, not the lack or presence
of version numbering.

In short:

a) if no DNG standard reader can understand the "private" stuff, why
bother storing it at all? In this case, DNG offers you nothing more
than what a TIFF file would.

Not true. For example, DNGs generated from the Raw files of my camera
have data that isn't understood by standard DNG readers, and it makes
no significant difference to me. I can't get at the lens model. If I
could, it might be possible to write a script to analyse all the lens
information in the metadata and apply automatic chomatic abberation
reduction. But, in the absence of that, I do it by eye.

I think this is the likely use of the private data. To give the camera
manufacturer's software opportunities to do things an easy way that
other processors have to do a harder way. (Without offering such a
"unfair advantage" to the camera manufacturers, it would be far harder
to persuade them to use DNG. Perhaps it is a "necessary evil").

DNG gives me, and others who use it, the advantages of Raw processing
over TIFF processing.

b) if some DNG reader _can_ understand it, it must be a custom reader,
and thus one has just converted the "file format problem" into an
"application format problem". If you have to keep track of reader X to
best deal with DNG's that come from camera X in order to get the most
out of the images, in what way would this be different from the current
situation? In this case, DNG has given you nothing but an extra, lazy,
man in the middle that drops bits on the floor.

Not true. See my example above. I agree that a formal standard for
archival purposes would be unlikely to have private data fields.
Proposals for an archival version of PDF, called PDF-A, don't have
encryption, for example. I can envisage an archival version of DNG,
perhaps DNG-A, that has no private bits.

Note that in both cases the DNG (or OpenRAW or whatever) gives you
nothing. Why, then, use it?

I have been using DNG since 10th October 2004. It gives me benefits,
more with each version of ACR, more with each version of the DNG
Converter, and lots more with the upgrade to CS2.

I get much smaller files, which makes processing faster and archiving
easier. I get my ACR settings and adjustments stored in the DNG file,
so that I have just one file to deal with. I have the confidence that I
will be able to process those archived DNG files for as long ahead as I
need, and I didn't have this confidence with the files directly from my
camera.

I know that ACR gives the same image whether I process the camera's
original files or DNG files. (I have tested Raw files for a number of
cameras of a number of makes, and in each case the result of the ACR
processing was the same for both the original Raw file and the DNG
version). If I buy a new camera after CS3 and ACR 4.x have been
released, I will probably not have to upgrade to CS3 to handle that new
camera.

The 3.1 DNG Converter stores the private data of my camera in the DNG
file. Although I can't currently do anything with it, at least it is
there. Perhaps in future I will be able to do something with it.
Perhaps a new DNG Converter will accept my current DNG files and move
that data to a place where it can be used directly. Who knows?

I think in future we will see high quality magazines, and other users
of photographs, accepting Raw files. I'm confident they will accept
DNG. They may sometimes insist upon it, just as now they may insist
upon TIFF 6.0.

DNG is opening up the world of Raw processing. I think we will see
benefits in future that we can't yet think of.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

Ryadia@Home wrote:
Barry Pearson wrote:
[snip]
The one thing you haven't mentioned here Barry is that RAW is not an
image format. Never has been, never will be. Photo Shop and all the
other apps I've tried including Canon's own RAW converter cannot save
back to a Camera raw file. It is essentially data in the process of
being used to "develop the film" to use your method of description.

Chuckle! Typical Raw files are based on the ISO standard "Tag Image
File Format / Electronic Photography". I doubt that you can sustain a
case that those Raw files are not image files! I suggest it is better
not to have a debate about terminology, because it will simply reduce
to a difference of opinion that cannot be resolved.

Nikon Capture 4.2.1 can save back to a NEF Raw file, and include within
it the settings you have just made. ACR 3.1 under Photoshop CS2 can
save its settings (eg. WB, noise, chromatic abberation reduction), and
adjustments (eg. crop, rotate, align, curves), back to a DNG file. Both
do so in a non-destructive way - they preserve the sensor data. There
is nothing inherent in Raw files that forces Canon to behave as it
does.

This is where I differ from the nerds and geeks. I see the data I
collect from the camera's sensor as Camera RAW, as just digital data.
Only after it is "converted" to an image format, can it considered an
image or even a digital negative.

You simply have a different definition of "image format" from me and
lots of others, including ISO. It isn't worth debating the terminology,
because we are unlikely to agree, and it doesn't matter anyway.

I think of "Digital Negative" simply as a brand, not a scientific or
engineering description. I guess, (but don't know), that Adobe thought
it would appeal to film photographers considering starting with
digital. The term was already in use, but rather more casually.

My argument for the camera makers is that they have a legal right to
protect their patents and products in what ever way they choose. Right
now I think they have been very accommodating in being so open as to
provide RAW data and they may well stop doing that if this push gets
going to any great strength which can be measured in $$$s.

I care about what is good for photographers and users of photographs.
Opening up the whole Raw processing industry and marketplace will be
good for us. I think it will be good for the camera manufacturers too,
except for the single matter that it will prevent them locking-in their
users to their own software.

Any top-end camera that doesn't provide Raw as an option will not sell
to serious amateurs and professionals. (Except perhaps to fast-action
photographers who may use JPEGs in order to get the required
write-to-card speed and size).

Use of a common Raw format such as DNG doesn't threaten patent
protection.

[snip]
I think if the nerds and geeks continue and the pressure gets great
enough for the camera makers (well Nikon anyway) to do something, they
may well decide to take Olympus's example and hide the whole bloody RAW
conversion process in the camera! That'd fix 'em right up!

Then they will lose lots of sales, see above. Raw offers far more than
TIFF 6.0 ever can, and with smaller file sizes in nearly all case!

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

On 28 May 2005 00:58:27 -0700, "Barry Pearson"
wrote:

This shows the power of DNG. You can write code to process files
conforming to particular versions of DNG, and you can cater
automatically for cameras that you haven't heard of, and that may not
have existed when you wrote the code. The advantage for the camera
manufacturer would be that their new camera is immediately supported by
all Raw processors that catered properly for the chosen version of DNG.
If a future D3X or 375D output in DNG version 1.1.0.0, then ACR 2.4
could process the images.

When new technology cannot be catered for by the current DNG version, a
new version, for example 2.0.0.0, will be needed. That presumably can't
be processed by ACR 2.4, nor other Raw processors. Upgrades will be
needed.

What I would like to see is more open source image editing systems.

Adobe Systems is making me very nervous with their attempt to
commoditze digital imagery. Once they have got us all using their
image processing system what is to stop them from hitting us at every
turn for more money. (Like CS2 and ACR 3.1).

From the Dave Coffin interview at dpreview
http://www.joelonsoftware.com/articl...gyLetterV.html