"Raw" file issues?

wrote:
[snip]
As I said, this will in many instances necessarily reveal trade secrets
or other facets of the technology that the manufacturers would likely
be unwilling to disclose. And if part of the OpenRAW is the signing of
an NDA, then can we honestly call it "open"?

Publicising current formats may reveal some things that manufacturers
would not like revealed. Although that may be just the design quality
of their Raw file formats! But interface specifications don't normally
reveal much about the inside of the box. I would expect the formats to
identify the sensor configuration, but the manufacturers are typically
willing, sometimes eager, to tell the world about that anyway.

Use of a common Raw format would reveal even less. At the moment, the
DNG format caters for Bayer sensors, Fujifilm SR sensors, Sony 4-colour
sensors, and Foveon sensors. (Possibly lots more configurations could
also be handled). If you can handle all of those with a single
specification, then using that specification isn't revealing much about
your technology. Formats designed specifically for your technology are
more likely to be revealing, if only accidentally.

[snip]
You are forgetting or ignoring the cost of innovations that are simply
unimplementable within the legacy framework. It is for this and other
reasons that the sensible person does not want camera manufacturers
constrained by some Adobe or OpenRAW or otherwise committee drafted
multi-volume 2356 page standard written in dense legalese.

What innovations are those? No new cameras in the last year and a half
would have needed a change to a common Raw format. The vast majority of
innovations are not visible in details of the Raw file format.

The DNG specification is 50 pages long, and a lot of that is white
space. It builds on TIFF/EP (the draft was 65 pages). But the camera
manufacturers used that anyway! Typical proprietary Raw formats are
already based on TIFF/EP. (And TIFF/EP refers to others). At the
moment, internally the manufacturers must use these external
specifications plus their own internal usage specifications. DNG, or
other common Raw format, would replace some internal specification.

Any photographer concerned with the future health of top-end digital
photography ought to want camera manufacturers to use a common Raw
format. And DNG is suitable for that purpose.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

wrote:
Alan Browne wrote:

Open RAW is best for everyone in the long term.

48V twisted pair POTS is a "standard" invented at the dawn of the
telephone era. There are many engineers who wish we could rip the
entire mess out of the ground, off the poles and start over again.
Offer praises to Allah or whoever that cell phones, VOIP and the rest
of it are doing the job indirectly.

Perhaps, without that standard, we would not be advanced enough to make
that wish! We would still be wasting too much time trying to get basic
communications to work.

We sort out one layer of the communications interworking. That enables
us to build lots of services in higher layers. That creates a desire
for better lower layers. The trick is to have an architecture, or
structural decomposition, that enable to make progress in one area
without having to tear up all other areas. And that needs clean
interfaces between the various areas.

If we DO decide to rip all that stuff out, I would hope we could do so
without (say) having to re-invent TCP/IP to run over its successor.

Be very careful what you wish for.

I would think that anyone who has spent many years or decades helping
to design complex multi-vendor systems knows what to wish for here! We
need a common Raw format. And in the meantime we need sufficient
publication of our current formats to build stop-gap measures such as
access to proprietary Raw formats, and conversion to a common Raw
format, etc.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

Barry Pearson wrote:

Publicising current formats may reveal some things that manufacturers
would not like revealed. Although that may be just the design quality
of their Raw file formats!

I can't comment on Nikon stuff, since I'm a Canon sort of person.

Canon 10D: a bizarre CIFF, for which they published the specifications
for the "superstructure" of this file. It's a kind of weird TIFF; you
can, in fact, TIFFize a CIFF. There are two images in this file: the
JPEG and an strangely encoded raw sensor dump. That Dave Coffin
managed to figure it out speaks for his abilities as a Reverse
Engineer.

Canon 1DMkII: it's a straight TIFF, with EXIF markup. Three images
embedded: a flat RGB thumbnail, a 1536 pixel wide JPEG (used for
in-camera editing), and the lossless JPEG encoded sensor dump.

The main mysteries for both are not the bulk format, but the
undocumented tags, specifically the "EXIF MakerNote" as well as the
various non-standard TIFF tags.

But interface specifications don't normally
reveal much about the inside of the box. I would expect the formats to
identify the sensor configuration, but the manufacturers are typically
willing, sometimes eager, to tell the world about that anyway.

I have reverse engineered a fair amount of the 10D's undocumented tags
(and some of the 1DMkII's -- too busy). There is one that does indeed
lay out of the geometry of the sensor (how large, where the "image"
starts, and so forth). There are other tags that detail the active AF
sensor bitmap, various light meter readings, exposure computations and
so forth. Just a few days ago, I found (by accident) the default
luminance curve in the 1DMkII CR2 file.

Probably the most interesting aspect is one that is relevant he the
sensor has a 64 column bitmap "black" pixels on the left side of the
image. The usual assumption is that these are used for bias level
stuff. However, there is unusual stuff occuring here when the
exposures grow to 30 seconds or more ... the 64 columns splits into
distinct 16 column set and a 48 column set, and these columns are
(apparently) used to produce a much better bias estimate than would be
obtained from simple averaging.

This is precisely the sort of detail that Canon would almost certainly
_not_ want to be generally known (it took me a while to figure it out),
not just that it exists, but that someone might be able to figure out
the device physics and go from there. If this is such a thing, though,
there are almost certainly others awaiting.

Or maybe everyone knows this stuff because it is a Canon patented
process.

Use of a common Raw format would reveal even less.

Suppose the above 16/48 column stuff was a Canon "trade secret" they
would prefer to keep. How could a standard format encompass such
sensor behaviour without revealing the secret? Or suppose that
everyone knows this -- it is a Canon patent. Unless Canon licenses
this patent for zero cost (which we can assume will be a
zero-probability event), no one can use it even if it was fully
documented to the last bit.

Formats designed specifically for your technology are
more likely to be revealing, if only accidentally.

I suggest this is more common than people think. The "MakerNote"
information is tightly coupled to the hardware, at least it is for
Canon equipment.

In article ,
Alan Browne wrote:
RichA wrote:

http://www.luminous-landscape.com/essays/raw-flaw.shtml

Let's all do our part!!

Well, I've started on my part.

Currently the Adobe DNG converter is pretty much the only choice
if you want to convert your RAW files to DNG. That's not a lot
of help for folks on a Unix/Linux platform, or for the putative
geek 25 years down the road trying to compile on his new platform.

So - I've registered a SourceForge project for an open source
RAW-to-DNG converter. I'd be interested in hearing from C++
programmers who could contribute to the effort.

I'd anticipate spending a week or so collecting contributors,
at which time I'll set up a mailing list and start discussions.

On 26 May 2005 13:10:49 -0700, "
wrote:

Alan Browne wrote:

wrote:

activity -- or even the media itself who regularly troll their audience
with prurient, irrelevant, or just plain false information.

Don't be an ass. Given the indications that companies like Nikon would
like nothing better than to sell more s/w as aprt of their camera
solution (other OEM's too), everyone has everything to gain by a
standard for RAW images.

It is not in Nikon's interest to tell everyone their innovations in
(say) automatic white balance, or in Canon's interest to spill their
beans (say) fancy device physics for optimal bias estimation in
long exposure images.

Reichman, et al, say they don't want "trade secrets" revealed, but they
are woefully clueless in that their demands for documentation of these
files is exactly that.

If you don't like Nikon's or Canon's policies about any of this, you
are free to purchase the products of other companies. Right? Or is
someone forcing you to purchase Nikon's software?

Really, what exactly is the problem here? If anything, the very
existance of Dave Coffin's "dcraw" makes the ranting Reichmann, et al,
look very kookish, and rendering the entire "OpenRaw" issue moot: RAW
file formats are as open as can be, _despite_ the best efforts of
Nikon.

Here we have an example of a prime case of the toadying
"company man" not concerned with customers. It's a disease
that afflicts many people who work in companies over 1000 people,
or those who fantasize that their own work is more valuable than
good customer service.
-Rich

In article om,
wrote:
Alan Browne wrote:

Ok, then, "OpenRAW has no point if the file formats can be cracked
easily".

You admit the file formats can be cracked easily, therefore ...

Since it is all but impossible to encrypt the data in the first place,
there is no need to do so.

Again, that is something for the manufacturers to decide. (Personally,
I agree with you and likely the guy who wrote the firmware for the D70,
D2x, etc is also in violent agreement. It's the dingbats in marketing,
etc, who are ignorant.)

Or I suppose the OEM's could begin keying
the encryption on a case by case basis. Yeah, uh huh.

As you know, it wouldn't matter. www.google.com: softice debugger (and
so forth).

If they wanted to be *serious* about the encryption, I don't
think that "softice" could do much about it.

Start with a public-key/private-key algorithm, similar to what
is ussed in PGP.

Embed the public key in the EXIF data, similar to the serial
number and model number (except that it would take significantly more
bytes of data).

Add a private key to get which you have to send to the vendor
both an image from your camera (to get the EXIF data), and a unique
number in the computer (hostID on Sun workstations, something else in
the firmware on a PC or a Mac). In exchange, you will get a key which
will work only with *your* camera on *your* computer. (It should be
possible to have the computer carrying the keys for several cameras, and
to recognize which is needed by examining the EXIF data. You could even
do whole image encryption.

But -- you have to be willing to pay the costs:

1) The file size will grow in the process of encryption.

2) The CPU usage in the camera would *greatly* increase -- cutting
into the frames per second rate in burst mode, and shortening
the battery life.

3) The camera will be useless to anyone else.


There are *some* benefits from this.

1) If you are working in a sensitive field, if the camera (or the
media) are stolen, the images are not compromised.

2) If the disk drives from the computer are stolen, that will
not be sufficient to use the images stored on it. (At least not
the RAW ones.)

3) If the camera is stolen, there will be minimal market for it,
without the key -- and to get another copy of the key from the
manufacturer, you would need to expose who now had the camera,
so stolen cameras would be easier to retrieve. (The
manufacturer would require the consent of the registered owner
to release new keys, so if you are not the registered owner, you
are likely the thief.)

Ideally, however, this sort of encryption would be turned off by
default, and would require an explicit action (similar to installing a
firmware patch) to turn it on. (Thus -- only those who feel that they
could benefit from the encryption would have to pay the costs listed
above.)

Under those circumstances, you could bypass the computer hostid
(or equivalent) part to the key, and supply private key and public key
in the same download -- to be separated in the computer which installs
the firmware in the camera. It could then modify itself (if so desired)
to lock to a single computer.

Enjoy,
DoN.
--
Email: | Voice (all times): (703) 938-4564
(too) near Washington D.C. | http://www.d-and-d.com/dnichols/DoN.html
--- Black Holes are where God is dividing by zero ---

John Francis wrote:


So - I've registered a SourceForge project for an open source
RAW-to-DNG converter. I'd be interested in hearing from C++
programmers who could contribute to the effort.

I'd anticipate spending a week or so collecting contributors,
at which time I'll set up a mailing list and start discussions.

I believe dcraw already reads DNG, if so, adding a write_DNG capability
should be relatively easy.

So considerer Dcraw.c as a beginning for the Guzinta depending on the
copyright provisions. See the GPL statement at top of
http://sunsite.rediris.es/pub/OpenBS...w-7.12/dcraw.c

From there, given the Adobe spec for DNG, the Guzouta should be
relatively easy.

Perhaps enlist Dave Coffin?

Cheers,
Alan.

--
-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
-- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
-- [SI] gallery & rulz: http://www.pbase.com/shootin
-- e-meil: Remove FreeLunch.

In article ,
DoN. Nichols wrote:

But -- I don't see how the situation with RAW files is any worse
than these limitations from the film days.

In the film days, you could buy a different (relatively cheap) film
to use in your (relatively expensive) camera. Nowadays the limitation
is embedded in the expensive component, and you can't avoid it.

In article ,
Alan Browne wrote:
John Francis wrote:


So - I've registered a SourceForge project for an open source
RAW-to-DNG converter. I'd be interested in hearing from C++
programmers who could contribute to the effort.

I'd anticipate spending a week or so collecting contributors,
at which time I'll set up a mailing list and start discussions.

I believe dcraw already reads DNG, if so, adding a write_DNG capability
should be relatively easy.

So considerer Dcraw.c as a beginning for the Guzinta depending on the
copyright provisions. See the GPL statement at top of
http://sunsite.rediris.es/pub/OpenBS...w-7.12/dcraw.c

From there, given the Adobe spec for DNG, the Guzouta should be
relatively easy.

Perhaps enlist Dave Coffin?

Cheers,
Alan.

Our goals are rather different. Dcraw reads just enough of the various
file formats (DNG included) to be able to produce converted images.

I'm far more interested in getting all that additional metadata from
the RAW files, and storing it in a publicly-documented format.

wrote:

Suppose the above 16/48 column stuff was a Canon "trade secret" they
would prefer to keep. How could a standard format encompass such
sensor behaviour without revealing the secret?

Does not the fact that DNG can *already* handle it indicate that this
"problem" is nonexistent? Does not the fact that DNG can fully handle
data from cameras that didn't exist at the time its code was written
mean that this has no actual basis in reality?

--
Jeremy |