"Raw" file issues?

Owamanga (not-this-bit) wrote:
[snip]
My copy of Word that come in a 2004 version of Office XP still opens
an ASCII text file format, a standard that dates back to 1963.

Today, the latest version of Photoshop still opens the HPGL/GCG files,
the oldest graphics format I can think of. Dating back to the early
80's or before. Postscript is nearly as old, and again, I can still
open those.

These are between 25 and 40 years old, and we can still open them. I
don't think people need to worry *quite* so much.
[snip]

Two points:

1. You name the products that can access those files. In the case of
PDF and ASCII text, there are lots, of course. But is that the case for
HPGL/GCG files? There is a risk that you start to get tied in to
particular products, running on particular platforms, to access these
old file formats.

2. A lot depends on the degree of motivation for products to support
particular formats. ASCII text and PDF are ubiquitous, so it is hardly
surprising that plenty of products can access them. And I think
Photoshop is better than many other products. (Did HPGL/GCG get a good
take-up?)

Can you properly print files created about 20+ years that used an
extension to the WordStar3 dot-commands and special control characters
to draw diagrams on a Ricoh 4120 (?) laser printer? No, of course you
can't! It was "proprietary". I devised that format myself, and wrote a
printer driver for WordStar3 to exploit it. A huge number of documents
in that format are now orphaned. Would it be satisfactory if I simply
provided the specification for those extensions, so that anyone could
write code to access them in future? No! If that was the only way to
save vital documents, it would have to done, but it is hardly the way
anyone would choose.

I have read of a couple of cases where Raw formats have already become
orphaned, or at least lost one parent. (I won't name them in case I
mislead). In one case, a 5-year old camera from a well known
manufacturer was dropped from the latest version of their leading tool,
leaving just their lesser tools to support it.

We have most chance of long life if:
- We have one or just a few formats of that type, instead of 100s.
- We have vastly many millions of files conforming to that format.
Preferably many millions per day.
- We have lots of products that support the format. Preferably
including open source code.

We have that for TIFF 6.0. We have it for JPEG. We need it for a Raw
format.

(I have an uneasy feeling that the camera manufacturers are far less
motivated to support their Raw formats for a long time than software
suppliers are).

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

On 27 May 2005 07:41:22 -0700, "Barry Pearson"
wrote:

Owamanga (not-this-bit) wrote:
[snip]
My copy of Word that come in a 2004 version of Office XP still opens
an ASCII text file format, a standard that dates back to 1963.

Today, the latest version of Photoshop still opens the HPGL/GCG files,
the oldest graphics format I can think of. Dating back to the early
80's or before. Postscript is nearly as old, and again, I can still
open those.

These are between 25 and 40 years old, and we can still open them. I
don't think people need to worry *quite* so much.
[snip]

Two points:

1. You name the products that can access those files. In the case of
PDF and ASCII text, there are lots, of course. But is that the case for
HPGL/GCG files? There is a risk that you start to get tied in to
particular products, running on particular platforms, to access these
old file formats.

Actually, post script is .PS, PDF is a newer format, no doubt they
have a lot in common, coming from the same people.

2. A lot depends on the degree of motivation for products to support
particular formats. ASCII text and PDF are ubiquitous, so it is hardly
surprising that plenty of products can access them.

Of course.

And I think Photoshop is better than many other products.

Definitely.

(Did HPGL/GCG get a good take-up?)

Err, I don't know enough about the other products available at the
time to comment really.

Can you properly print files created about 20+ years that used an
extension to the WordStar3 dot-commands and special control characters
to draw diagrams on a Ricoh 4120 (?) laser printer?

Sure. If I need to. No doubt the printer commands were fairly basic. A
quick gander at the file and I can see how things hang together. But
it comes down to effort vs reward (motivation) as you've mentioned.

No, of course you
can't! It was "proprietary". I devised that format myself, and wrote a
printer driver for WordStar3 to exploit it.

Well, no wonder you've experienced problems. Your home-grown
proprietary format that nobody else knew about - how would you
*expect* someone like microsoft to support it?

Nikon are chucking out D70s at a rate of around 90,000 units a month,
just one model from their digital line up. If each user shoots 6,000
per year (not difficult) then in just two years of sales we've got 6
billion files which I believe is motivation enough for anything).

(Of course, not everyone shoots RAW..)

So, I am being selfish and only worrying about me, I produce NEF files
and CRW files and am fully confident that those two formats will be
supported in graphics software for as long as I'll need them. The day
I realize this belief is misplaced is the day I'll convert them to
something more suitable (*). Computers of the future will of course be
so powerful that it'd already have done this in the background just in
case I wanted it to do that.

g

(*) - This would need to be some form of open RAW, such as of course,
the Adobe one discussed in the article.

g

A huge number of documents
in that format are now orphaned. Would it be satisfactory if I simply
provided the specification for those extensions, so that anyone could
write code to access them in future? No! If that was the only way to
save vital documents, it would have to done, but it is hardly the way
anyone would choose.

It's a very specific example. (but I did ask I guess)...You can use
the same agrement to try and stop the French speaking French. In 100
years, nobody is going to be able to understand that stuff anyway, so
why bother?

I have read of a couple of cases where Raw formats have already become
orphaned, or at least lost one parent. (I won't name them in case I
mislead). In one case, a 5-year old camera from a well known
manufacturer was dropped from the latest version of their leading tool,
leaving just their lesser tools to support it.

Hey it's the net, just add a disclaimer that you *think* it was so and
so, people will soon put you straight, and no damage is done.

We have most chance of long life if:
- We have one or just a few formats of that type, instead of 100s.
- We have vastly many millions of files conforming to that format.
Preferably many millions per day.
- We have lots of products that support the format. Preferably
including open source code.

We have that for TIFF 6.0. We have it for JPEG. We need it for a Raw
format.

We don't have it for TIFF. TIFF is just a container, like AVI, there
is no guaranteed backwards compatibility. Software that read TIFFs 5
years ago may not be able to read a TIFF created today, we just hope
the opposite isn't the case (take a look at the LZW/GIF wars for
example).

Same is true with JPEG2000, at least for some of it's features.

(I have an uneasy feeling that the camera manufacturers are far less
motivated to support their Raw formats for a long time than software
suppliers are).

Probably true, but we don't care whether camera manufacturers support
it *as long as the software suppliers do*. The encryption stuff they
are pulling is nonsense, I agree. They didn't think about the
ramifications and I am confident it will either go away in the future,
or they'll make no aggressive legal moves to attempt to prevent 3rd
party software vendors from reading the format.

--
Owamanga!
http://www.pbase.com/owamanga

Alan Browne wrote:
[snip]
Hence my big yawn over DNG ... until I discovered the other day in an
actual experiment with real files that DNG takes 26 to 41% less space
than .MRW. So I'm mulling the transition to DNG solely to reduce the
DVD count by 1 for every 3 or 4 DVD's. I haven't committed to the
decision yet.

But prior to that, my DNG 'objection' was: when the OEM's adopt it, I'll
adopt it.

You illustrate why there is so much disagreement over DNG. Whether
someone benefits from DNG depends on the camera used, the workflow, the
version of Photoshop used, what package is used for Raw processing, the
perception of risk and impact of orphaned Raw formats, etc. (The
balance shifts month by month, always towards more likely benefit from
DNG).

If you used the 350D, you would have discovered that the saving was
only about 6% to 7%. (Yawn!) You probably wouldn't even be mulling!

But if you used the Pentax *istD, you would long ago have discovered
that the saving was well over 50% (from about 13MB to about 6MB). And
if, like me, you didn't have a DVD burner, but archived to CD.... You
would already be a very enthusiastic user of DNG!

Here is something I wrote elsewhe

1. Someone who uses a Raw processor that won't accept DNG will get
little or no benefit from DNG. They may decide to archive a DNG version
of their Raw files to increase the chance that they will be able to
read the files years later. But they will have to work with the
original Raw file. Users of the cameras' own software (currently) come
into this category.

2. Users of ACR 2.x under CS may get significant, or little, or no,
benefit from DNG, depending on their workflow. I benefited, and someone
with a camera only supported by ACR 3.1 / 3.1 DNG Converter, such as
the 350D or D2X, can benefit a lot. But others may not benefit, or only
a little.

3. Users of ACR 3.1 under CS2 are likely to get significant benefit
from DNG. This is the release where its benefits have become much more
obvious. Since upgrading, I have started to convert to DNG straight
from the card without an embedded version of the original file.
(Against the advice of Adobe). Being able to hold ACR 3.1 settings and
adjustments within the DNG file in a non-destructive way makes file
management easier. Obviously I get the benefits of the smaller file
size.

It is not very surprising that a lot of people can't yet benefit, and
lots of people remain to be convinced. DNG was launched 8 months ago
TODAY. That is not long, given the things that have to come together
before most people can benefit. I wonder how many people were talking
about TIFF 8 months after it was launched? PostScript? PDF? HTML? CSS?
(SVG!)

I would advice everyone to investigate DNG, and make up their own mind
about whether it suits them. But, as noted above, it certainly won't
suit everyone.

Yet!

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

Jeremy Nixon wrote:

Suppose the above 16/48 column stuff was a Canon "trade secret" they
would prefer to keep. How could a standard format encompass such
sensor behaviour without revealing the secret?

Does not the fact that DNG can *already* handle it indicate that this
"problem" is nonexistent?

Cite the relevant sections in the DNG specification which detail
exactly this -- that is, how to handle the 16/48 column stuff -- and
you have made a point.

In article ,
Owamanga wrote:
On Fri, 27 May 2005 01:20:05 +0000 (UTC), (John
Francis) wrote:

In article ,
Alan Browne wrote:
RichA wrote:

http://www.luminous-landscape.com/essays/raw-flaw.shtml

Let's all do our part!!

Well, I've started on my part.

Currently the Adobe DNG converter is pretty much the only choice
if you want to convert your RAW files to DNG. That's not a lot
of help for folks on a Unix/Linux platform, or for the putative
geek 25 years down the road trying to compile on his new platform.

So - I've registered a SourceForge project for an open source
RAW-to-DNG converter. I'd be interested in hearing from C++
programmers who could contribute to the effort.

C++ ? Are you unix peeps still using that?

You'd prefer DECSystem-10 assembly language? Or Apollo Pascal?
I'll stick with what I know.

All joking aside, one of the goals is to ensure longevity of
the project. In 25 years time C++ will probably be close to
what it is today - there's just too much C/C++ code out there
for it to go away. I'm not sure the same can be said of any
other language. I expect Java, for example, to undergo some
radical changes - new embedded systems turn up every day, with
new requirements that can push the boundaries of the language.

Barry Pearson wrote:

By "weird TIFF", do you mean "weird TIFF 6.0" or "weird TIFF/EP"?

I mean "weird TIFF", as "unusual", "not exactly the same as but similar
to in strange way", or "derived from by those under the influence of
unspecified narcotic agents", and so forth. It has a directories,
tags, and the rest of it. Everything else is completely incompatible
with TIFF as is, but it can be easily converted into whatever TIFF one
wishes (if anyone was so inclined).

But it also defines how camera manufacturers can safely store any "secret
sauce" in a DNGPrivateData tag defined for the purpose. "Safely" means
that anyone else can re-write a DNG file while preserving the private
data, because although its contents are private, its format is
well-structured.

You need to read the ranting of the OpenRAW people. They explicitly
demand "no secret sauce" -- though irrationally maintaining they are
not making such a demand.

Barry Pearson wrote:

SNP

But if you used the Pentax *istD, you would long ago have discovered
that the saving was well over 50% (from about 13MB to about 6MB). And
if, like me, you didn't have a DVD burner, but archived to CD.... You
would already be a very enthusiastic user of DNG!

I suggest you become an enthusiastic user of DVD too.

Here is something I wrote elsewhe

SNP

Everyone has to keep updated and abreast of changes, to be sure. But
regarding DNG (or some standard 'RAW') the OEM's really need to navel
gaze a bit and do what's right for the customers, not themselves.

Cheers,
Alan

--
-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
-- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
-- [SI] gallery & rulz: http://www.pbase.com/shootin
-- e-meil: Remove FreeLunch.

wrote:

Cite the relevant sections in the DNG specification which detail
exactly this -- that is, how to handle the 16/48 column stuff -- and
you have made a point.

It doesn't have to specify such a thing. That's not what DNG is, or what it
does. It doesn't have to handle, or know about, every technical difference
in every sensor design. Indeed, it proves that using a different RAW format
for every camera model is completely unnecessary.

--
Jeremy |

Jeremy Nixon wrote:

Cite the relevant sections in the DNG specification which detail
exactly this -- that is, how to handle the 16/48 column stuff -- and
you have made a point.

It doesn't have to specify such a thing.

Yes it does. If the native camera format can do things that the DNG
can't, why use the DNG?

That's not what DNG is, or what it does.

Shifting the goal-posts now? OpenRAW wants full, complete,
documentation.

It doesn't have to handle, or know about, every technical difference
in every sensor design.

If it doesn't, then you lose capabilities that may be associated with
the sensor. If the DNG removed the red channels from all RAW images,
would you use it?

Indeed, it proves that using a different RAW format
for every camera model is completely unnecessary.

I have data that suggests that Canon cameras have unique black level
estimation for long exposure images. The DNG format does not allow for
such behaviour (or at least you have not presented me with a citation
-- and I couldn't find one when the DNG spec was initially released).

Now if the DNG can't handle that, what do you recommend I do with my
multi-minute minute exposure CR2 files I have?

wrote:

It doesn't have to specify such a thing.

Yes it does. If the native camera format can do things that the DNG
can't, why use the DNG?

It's not something the format, native or DNG, "does".

It doesn't have to handle, or know about, every technical difference
in every sensor design.

If it doesn't, then you lose capabilities that may be associated with
the sensor.

No, you don't.

I have data that suggests that Canon cameras have unique black level
estimation for long exposure images. The DNG format does not allow for
such behaviour (or at least you have not presented me with a citation
-- and I couldn't find one when the DNG spec was initially released).

Now if the DNG can't handle that, what do you recommend I do with my
multi-minute minute exposure CR2 files I have?

It is not something DNG needs to "handle".

Try taking your files, converting them to DNG, and loading them into
Camera Raw. It'll work.

--
Jeremy |