"Raw" file issues?

Alan Brownbe wrote:
Jeremy Nixon wrote:

Canon has already dropped support for the RAW files from one of their
cameras, in their new software.

If you're going to drop message like that then specifics are in order.

Sorry, I thought Canon dropping support for the D30 in new updates was
common knowledge.

You can still get the old software, of course, which will still run on
current computers.

--
Jeremy |

Owamanga wrote:

You could extend this argument and suggest that hardware manufacturers
aught to turn over the API and technical docs so we can all write our
own firmware, because god forbid, they one day stop doing firmware
updates for our old fossils.

No, that has no relation to what we're talking about. That would deal
with new pictures you'd take at some future point. I'm talking about
pictures you took two years ago. What happens when the camera maker
decides to stop supporting your old camera, and the old software stops
working on current computers?

You use third-party software, of course, and read them just fine. So
why in the name of all that's good and pure would you want the camera
maker to try to prevent that, guaranteeing that your old files will be
useless in the above scenario?

--
Jeremy |

Jeremy Nixon wrote:

Sorry, I thought Canon dropping support for the D30 in new updates was
common knowledge.

Why would it be?

If this is so, it's a great argument for Canon users to pressure Canon
to use an open format like DNG and not worry about support for the future.

Cheers,
Alan.

Alan Brownbe wrote:
Jeremy Nixon wrote:

Sorry, I thought Canon dropping support for the D30 in new updates was
common knowledge.

Why would it be?

Because it wasn't kept secret, was discussed here (well, on r.p.d) when it
happened, and the information is pretty widely available.

--
Jeremy |

Barry Pearson wrote:

Frankly, I haven't a clue whether the "unique black level estimation
for long exposure images" is catered for by the current version of DNG.

It isn't.

Since I don't use Canon, I have no need to think about it.

www.google.com: define:abstract

Nikon (or whoever) has likely has something that is being dropped on
the floor when you convert to DNG. This is the basic point (that is
apparently lost on too many people): without specific support for all
features present in a camera, DNG is necessarily lossy.

Why couldn't
it be held in DNGPrivateData, or in MakerNote with MakerNoteSafety set
to 1 (safe)?

Because no standard DNG reader will know what to do with it, why
bother?

If it can't, then Canon could ask for a new version.

Why should they? Instead they can make a special reader which does
know what to do with the "PrivateData" ... but hey, wait a minute ...

That is why DNG has a good version scheme.

The existence of "PrivateData" is the problem, not the lack or presence
of version numbering.

In short:

a) if no DNG standard reader can understand the "private" stuff, why
bother storing it at all? In this case, DNG offers you nothing more
than what a TIFF file would.

b) if some DNG reader _can_ understand it, it must be a custom reader,
and thus one has just converted the "file format problem" into an
"application format problem". If you have to keep track of reader X to
best deal with DNG's that come from camera X in order to get the most
out of the images, in what way would this be different from the current
situation? In this case, DNG has given you nothing but an extra, lazy,
man in the middle that drops bits on the floor.

Note that in both cases the DNG (or OpenRAW or whatever) gives you
nothing. Why, then, use it?

wrote:

Nikon (or whoever) has likely has something that is being dropped on
the floor when you convert to DNG. This is the basic point (that is
apparently lost on too many people): without specific support for all
features present in a camera, DNG is necessarily lossy.

Except that this is not correct. DNG does *not* need specific support
for all camera features in order to preserve them; indeed, that's part
of the whole point of the thing.

--
Jeremy |

Jeremy Nixon wrote:
[snip]
I'm currently processing my camera's RAW files after converting to DNG.
My camera didn't exist at the time the version of Camera Raw I'm using
was written. Yet, somehow, it works perfectly. How do you suppose that
is? How could DNG possibly "know about" my camera's unique RAW format?

Because it doesn't have to, of course.
[snip]

Thomas Knoll told me that no camera of the last year and a half would
have needed a change to DNG, had DNG existed that long ago.

Typically, to support a new camera, it isn't the DNG specification that
needs to change. Instead, what has to happen is that something has to
write, into specified places in the DNG file, specific information
about the camera model. (In addition to the sensor data for the
photograph, of course!) Then the Raw processing software, ACR or
something else, can extract this specific information instead of having
built-in knowledge of the camera.

I believe it works like this. Suppose I launch a new camera tomorrow,
that writes Raw files with a BCP extension. I also release some
software to process those Raw files. The camera has some existing
sensor configuration, such as Bayer, or Fujifilm SR, or Sony 4-color,
or Foveon, and writes just the sensor data to the BCP file. My software
obviously knows what the sensor configuration is, and interprets the
sensor data accordingly. This also includes interpreting the sensor
values according to the colour temperature.

No other software has a clue how to interpret the sensor data. It won't
even know what the sensor configuration is, and that is obviously
vital, even before you worry about the colour temperature and other
things. All Raw processing software will there have to give up on this
BCP file format. Obviously, ACR 2.4 in Photoshop CS has to give up,
because its built-in tables don't cover this camera.

Now Adobe releases a new version of the DNG Converter that knows about
this camera. This DNG Converter writes the sensor data to the DNG file,
of course. And also writes fields that describe the sensor
configuration, and how to interpret the colour temperature, etc. So now
the DNG file is "self contained". Some fields provide the sensor data
for the photograph. Others identify how to interpret that data.
(Metadata).

Now, when ACR 2.4, or any other Raw processor that handles DNG, reads
that file, it doesn't need its own data to tell it how to interpret the
data for this camera. It just extracts that from the DNG file. It can
handle cameras it has never before come across, because the DNG file
itself tells it how to. (I think the DNG Converter writes colour
calibration fields for 2 colour tempertaures, one tungsten and the
other daylight, to the file. This enables ACR to do its white balance
handling).

I'm still hazy about some of the details. But what I've said above is
consistent with everything I've learned and tested so far. I have
converted D2X and 350D Raw files into DNG, and processed them in ACR
2.4 under CS. CS Browser shows thumbnails for the DNG files just as it
would any Raw files that it recognised, and ACR 2.4 can process them.
But it just shows place-holders for the original Raw files, and refuses
to open the latter because it doesn't recognise the format.

It is this concept of being "self contained", with the sensor data for
the image, plus metadata identifying how to interpret the sensor data,
that makes DNG very powerful. Someone could release a photo-editor
tomorrow that only accepted DNG, and it would be able to process all 77
cameras that the DNG Converter could handle. Someone could release a
new camera tomorrow that recorded DNG, and immediately all Raw
processors that handled DNG properly would support that camera. And in
decades to come, DNG processors will be able to handle old DNG files
that have images taken using cameras that no one knows about anymore.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

In article .com,
wrote:
DoN. Nichols wrote:

If they wanted to be *serious* about the encryption, I don't
think that "softice" could do much about it.

Cryptographic snake-oil. Numerous people have proposed mechanisms that
can somehow secure information on fundamentally insecure/untrusted
hardware or media. None of them have worked. The RIAA, MPAA and other
"content" people have idiotic, aphysical dreams about this, but it
won't ever happen, no matter how many lawsuis they file, laws they
manage to pass, etc.

These are simple encryption designed to be fast to accomplish,
on flowing data, and are inherently easy to reverse. They are not what
I would refer to by the phrase above "serious about encryption". A
whole five minute audio cut would probably take an hour or more to
decrypt.

Add a private key to get which you have to send to the vendor
both an image from your camera (to get the EXIF data), and a unique
number in the computer (hostID on Sun workstations, something else in
the firmware on a PC or a Mac). In exchange, you will get a key which
will work only with *your* camera on *your* computer.

The camera maker _may_ be able to trust the camera, but that's as far
as they can go. Everything else is not under their control.
Extracting the final crypto-key would be a simple debugging job.

Hmm ... how much do you know about public-key/private-key
systems? The only key in the camera would be the public key, and they
could post it on the net for all the aid this would give to breaking it.
Both keys are very large primes, and it takes *both* to go from original
data to final data.

Very large primes are computationally difficult to extract from
the encrypted data and the known public key -- enough so that even large
arrays of high-powered computers will take days to months to accomplish
the task.

The assumption here is that each owner who desires it receives a
matching public and private key (different from all of those that all
other users receive). And it should be *optional* as to whether
encryption would be used or not -- it would be for the benefit of the
owner (and/or his employer), not the camera manufacturer. The default
should be no encryption, and it should require a complex series of
actions to enable it, given the disadvantages which I listed earlier.

I'm not suggesting this for a manufacturer to protect the white
balance data or something simple like that -- but for people who need to
take photos of classified objects, where the loss of the camera or the
media would result in risks to the country. Thus, the owner of the
computer (which would have the private key) would have every incentive
to protect that key (and the computer). You may not know this, but it is
illegal to connect a computer which is processing (or which *has* once
processes classified material to the internet. It *can* be connected to a
classified network (all systems handling the same classification level).

This is *not* the same thing as a manufacturer protecting some
part of their image format by simple-minded encryption -- this is
*serious* encryption that I am talking about -- and it would probably
take several seconds (or perhaps minutes) to encrypt each image. (And
the camera would *not* have the information needed to subsequently
*decrypt* the image.) Depending on the kind of data, and its
classification level, the jpeg thumbnail embedded in the RAW image could
be allowed to remain so chimping could verify the results, since it has
far lower resolution.

Enjoy,
DoN.
--
Email: | Voice (all times): (703) 938-4564
(too) near Washington D.C. | http://www.d-and-d.com/dnichols/DoN.html
--- Black Holes are where God is dividing by zero ---

Barry Pearson wrote:

Snipped

This systematic coupling of innovative hardware, firmware, and software
components via layers of standards and agreed specifications under
version control is the normal way our complex multi-vendor world works.
But ... I don't shoot TIFF or JPEG. I shoot Raw. Are we supposed to
believe that these principles can't work for Raw formats? That if a
magazine wants my Raw image, I shouldn't give them a standard format,
but I should give them the proprietary specification so that they can
obtain software to extract the image? Of course not! It is obvious that
the same principles can and should apply to Raw formats too.

Raw is not different *in principle* from all the other components. It
simply hasn't yet reached the same level of maturity. Those of us who
realise its immaturity need to be encouraging it to grow up. We musn't
let it believe that it can remain adolescent for ever. We certainly
shouldn't be making excuses for it!

Photographers, and users of photographs, need future Raw formats to
conform to an agreed specification, which will become at least a
de-facto standard. Then we need a way for older Raw files to be
convertable satisfactorily to the agreed format, so that they can be
handled in the same way without risk of being neglected. The primary
reason for obtaining the specifications for those older Raw formats
should be to enable high-quality converters to be developed. NOT so
that all packages that ever handle Raw files recognise all those Raw
formats for ever. Expecting (say) a new Raw processing package launched
in 5 years time to support 100s of Raw formats is just plain silly! Why
should it have to recognise more than 1?

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/


The one thing you haven't mentioned here Barry is that RAW is not an
image format. Never has been, never will be. Photo Shop and all the
other apps I've tried including Canon's own RAW converter cannot save
back to a Camera raw file. It is essentially data in the process of
being used to "develop the film" to use your method of description.

This is where I differ from the nerds and geeks. I see the data I
collect from the camera's sensor as Camera RAW, as just digital data.
Only after it is "converted" to an image format, can it considered an
image or even a digital negative.

My argument for the camera makers is that they have a legal right to
protect their patents and products in what ever way they choose. Right
now I think they have been very accommodating in being so open as to
provide RAW data and they may well stop doing that if this push gets
going to any great strength which can be measured in $$$s.

Olympus (the digicam my wife has) has a TIFF capture for their idea of
RAW data. Now this is an image file and it can be edited direct from the
camera and saved straight back to the same file or posted unaltered for
viewing with your "standard" viewers.

If Olympus have bypassed all the nerds and geeks arguments by providing
all the RAW data conversion in the camera, they have managed to keep
their patented process secret and intact and still provide the
photographer with a "digital Negative"... That make sense although many
will still push their barrow to try and force open source principals
onto a closed source environment.

I think if the nerds and geeks continue and the pressure gets great
enough for the camera makers (well Nikon anyway) to do something, they
may well decide to take Olympus's example and hide the whole bloody RAW
conversion process in the camera! That'd fix 'em right up!
--
Douglas...
It's traditional, painter's use it, Rembrandt used it.
Now you can put your photos on it too!
http://www.canvasphotos.com.au

Jeremy Nixon wrote:

Nikon (or whoever) has likely has something that is being dropped on
the floor when you convert to DNG. This is the basic point (that is
apparently lost on too many people): without specific support for all
features present in a camera, DNG is necessarily lossy.

Except that this is not correct. DNG does *not* need specific support
for all camera features in order to preserve them; indeed, that's part
of the whole point of the thing.

Even taken alone, your statement is nonsensical. It is isomorphic to
"You don't need an arm in order to use an arm."

Start making sense, dude.