'extra' lossless compression for camera raw images

"Paul Furman" wrote in message
. ..
Sachin Garg wrote:
"ray" wrote in message
...
On Thu, 20 Mar 2008 09:40:05 +0530, Sachin Garg wrote:

"ray" wrote in message
...
On Thu, 20 Mar 2008 08:52:46 +0530, Sachin Garg wrote:

"ray" wrote in message
...
On Wed, 19 Mar 2008 18:08:26 -0400, me wrote:

On 19 Mar 2008 21:49:36 GMT, in rec.photo.digital ray
wrote:

I'd be happy to do that if you'd provide specific information on a
couple of images. Looking at the bar graphs is really not
suitable.
The bar graphs are for the images he provides on the site.
Understood. And I'd be happy to use several compression programs to
do the same ones - if I had any results to compare to. BTW - I've
just done a simple bzip2 compression on a Kodak P850 raw file -
compressed size is 53% of uncompressed.
Thats great. If even bzip2 is able to get 53%, it will be very
interesting to see how much better sgraw will be.
Yes, it certainly would. It gives "Error: unsupported file format" -
bzip2 works properly.
Yep, for now sgraw only supports Nikon and Fuji raw formats. I will
definitely try to support kodak formats in future, but for now they are
unsupported.

Really. I would have thought you would have started with DNG - it is
quite well documented.

Yep, its documented and I have tried (with partial success till now).

Problem is that as DNG files need to support raw files from all cameras,
there are lots of minor camera-specific variations in it depending on
which orginal raw format a DNG was created from (a reason why editing
softwares aren't always able to support DNGs from cameras they don't
support natively).

So supporting all DNGs will mean supporting all those variations. Now
these are 'small' variations, nothing too difficult, but will need
cosiderable time and effort due to huge no. of cameras.

Get some programmers working on it, pay them with licensing fees from
Adobe grin.

LOL

Seriously, I convert to dng & would like this but I'm not going to archive
my photos in an obscure third party format.

Yes, thats understandable.

It might help a little to consider that conversion with sgraw is two-way. We
will always be able to convert back to original formats, so there will be
"no vendor lock-in" which is what usually makes obscure 3rd party formats
ugly.

If it can be done fast on-camera, the camera manufacturers could use it.

Its fast, but looking at what camera makers use right now, and the speed
expectation from cameras (multiple shots per second), it will probably be a
little while before this can be seen in camera (and thats just the technical
side of trouble).

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

On 19 Mar 2008 16:47:38 GMT, ray wrote:

On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:

I just finished writing a small tool (sgraw) which can losslessly
compress camera raw images, usually between 20-to-60%. (Regulars here
might recall I have mentioned this 'possibility' earlier, I finally put
in some hours to actually build a working demo :-)

http://www.sachingarg.com/compression/sgraw/

That would be savings between 1GB to 3GB for every 5GB of data.

Even when camera has already compressed the raw image (which isn't
always the case), its still possible to squeeze out extra compression as
cameras have to use simple algorithms for speed.

Compression is totally lossless, we get back the exact file we start
with (just like with zip), so both pixel information and meta-data are
perfectly restored.

This is just the first quick hack version (and supports all nikon and
fuji cameras), so it obviously wouldn't make sense to actually use it
for backups etc, but if there is enough interest I can build it further
to support other raw formats, along with improved compression and speed.

Your thoughts on this?

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

It seems totally superfluous since there are already dozens of
compression programs available. What makes you think you've discovered
some 'new and improved' algorithm?

Agreed. I just used WinRAR on a couple of RAW files and got anywhere from 38% to
52% compression rates by just using its auto defaults. RAR is a well-known and
universal compression method that works well for all multimedia formats. Unless
you can consistently do greater than 50-60% on all RAW files it's not worth
anyone's time to test this for you. Well, except as your own as a learning
exercise and a way to get attention for wasting everyone else's time and
resources in doing so.

"KandorMan" wrote in message
...
On 19 Mar 2008 16:47:38 GMT, ray wrote:

On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:

I just finished writing a small tool (sgraw) which can losslessly
compress camera raw images, usually between 20-to-60%. (Regulars here
might recall I have mentioned this 'possibility' earlier, I finally put
in some hours to actually build a working demo :-)

http://www.sachingarg.com/compression/sgraw/

That would be savings between 1GB to 3GB for every 5GB of data.

Even when camera has already compressed the raw image (which isn't
always the case), its still possible to squeeze out extra compression as
cameras have to use simple algorithms for speed.

Compression is totally lossless, we get back the exact file we start
with (just like with zip), so both pixel information and meta-data are
perfectly restored.

This is just the first quick hack version (and supports all nikon and
fuji cameras), so it obviously wouldn't make sense to actually use it
for backups etc, but if there is enough interest I can build it further
to support other raw formats, along with improved compression and speed.

Your thoughts on this?

It seems totally superfluous since there are already dozens of
compression programs available. What makes you think you've discovered
some 'new and improved' algorithm?

Agreed. I just used WinRAR on a couple of RAW files and got anywhere from
38% to
52% compression rates by just using its auto defaults. RAR is a well-known
and
universal compression method that works well for all multimedia formats.

Yes, general purpose compressors like rar/bzip2/zip etc can give good
results if original raw format did not use any compression (lossless). sgraw
is tuned for raw images, so it can give consistently better results than
them (theoritically at better speed too, but current implementation is slow.
I will try to improve this).

Besides, it can further compress the raw files even if they were already
compressed. Compression ratio is relatively less in this case as there are
technical limitations.

I would love to see any results different from what I found in my testing.
Which camera's raw files did you use for your tests? If these were from a
Nikon or Fuji, I am hoping that you will be amazed to see the difference
when using sgraw. If it was some other camera then we will have to wait till
I can work on those formats.

Unless
you can consistently do greater than 50-60% on all RAW files it's not
worth
anyone's time to test this for you. Well, except as your own as a learning
exercise and a way to get attention for wasting everyone else's time and
resources in doing so.

Fortunately, not everyone uses 'all' the cameras :-) If its getting results
on your raw files which are good enough for you, I will be happy to know
that you can find it useful. Otherwise, my apologies for wasting your time.
(If rar is getting those results you mentioned on your raw files, I am very
optimistic that you will like what sgraw will manage to do for you :-)

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

On Mon, 24 Mar 2008 10:13:05 +0530, Sachin Garg wrote:

"KandorMan" wrote in message
...
On 19 Mar 2008 16:47:38 GMT, ray wrote:

On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:

I just finished writing a small tool (sgraw) which can losslessly
compress camera raw images, usually between 20-to-60%. (Regulars here
might recall I have mentioned this 'possibility' earlier, I finally
put in some hours to actually build a working demo :-)

http://www.sachingarg.com/compression/sgraw/

That would be savings between 1GB to 3GB for every 5GB of data.

Even when camera has already compressed the raw image (which isn't
always the case), its still possible to squeeze out extra compression
as cameras have to use simple algorithms for speed.

Compression is totally lossless, we get back the exact file we start
with (just like with zip), so both pixel information and meta-data
are perfectly restored.

This is just the first quick hack version (and supports all nikon and
fuji cameras), so it obviously wouldn't make sense to actually use it
for backups etc, but if there is enough interest I can build it
further to support other raw formats, along with improved compression
and speed.

Your thoughts on this?

It seems totally superfluous since there are already dozens of
compression programs available. What makes you think you've discovered
some 'new and improved' algorithm?

Agreed. I just used WinRAR on a couple of RAW files and got anywhere
from 38% to
52% compression rates by just using its auto defaults. RAR is a
well-known and
universal compression method that works well for all multimedia
formats.

Yes, general purpose compressors like rar/bzip2/zip etc can give good
results if original raw format did not use any compression (lossless).
sgraw is tuned for raw images, so it can give consistently better
results than them (theoritically at better speed too, but current
implementation is slow. I will try to improve this).

Besides, it can further compress the raw files even if they were already
compressed. Compression ratio is relatively less in this case as there
are technical limitations.

I would love to see any results different from what I found in my
testing. Which camera's raw files did you use for your tests? If these
were from a Nikon or Fuji, I am hoping that you will be amazed to see
the difference when using sgraw. If it was some other camera then we
will have to wait till I can work on those formats.

Unless
you can consistently do greater than 50-60% on all RAW files it's not
worth
anyone's time to test this for you. Well, except as your own as a
learning exercise and a way to get attention for wasting everyone
else's time and resources in doing so.

Fortunately, not everyone uses 'all' the cameras :-) If its getting
results on your raw files which are good enough for you, I will be happy
to know that you can find it useful. Otherwise, my apologies for wasting
your time. (If rar is getting those results you mentioned on your raw
files, I am very optimistic that you will like what sgraw will manage to
do for you :-)

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to convert
back.

On Mon, 24 Mar 2008 16:31:25 -0400, me wrote:

On 24 Mar 2008 16:02:42 GMT, in rec.photo.digital ray
wrote:


Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to convert
back.

You can't, as DNG doesn't preserve 100% of the metadata.

Of course I can.

wrote:
On 24 Mar 2008 22:38:33 GMT, in rec.photo.digital ray
wrote:

On Mon, 24 Mar 2008 16:31:25 -0400, me wrote:

On 24 Mar 2008 16:02:42 GMT, in rec.photo.digital ray
wrote:


Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to convert
back.
You can't, as DNG doesn't preserve 100% of the metadata.
Of course I can.

Not according to Adobe in:

Introducing the Digital Negative specification: Information for
Manufacturers:
http://www.adobe.com/products/dng/pd...ufacturers.pdf

"It should be noted that the Adobe DNG Converter will not necessarily
maintain all of the private metadata in certain camera-specific raw formats
because this information
is not publicly documented and therefore not available to Adobe. However,
the Adobe DNG Converter will maintain all of the original image data as
well as all of the metadata needed for a high-quality final conversion.
Arguably, the private metadata is not really archival, regardless of the
format used, simply because it is undocumented. Nevertheless, Adobe
recommends that, when photographers use the Adobe DNG Converter for
archival purposes, they should maintain both the resulting DNG file and the
original camera-specific file."

Note the words "not necessarily maintain....", certainly leaving room
for some RAW formats that have all data captured regardless.

--
John McWilliams

On Mon, 24 Mar 2008 19:15:29 -0400, me wrote:

On 24 Mar 2008 22:38:33 GMT, in rec.photo.digital ray
wrote:

On Mon, 24 Mar 2008 16:31:25 -0400, me wrote:

On 24 Mar 2008 16:02:42 GMT, in rec.photo.digital ray
wrote:


Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to
convert back.

You can't, as DNG doesn't preserve 100% of the metadata.

Of course I can.

Not according to Adobe in:

Introducing the Digital Negative specification: Information for
Manufacturers:
http://www.adobe.com/products/dng/pd...ufacturers.pdf

"It should be noted that the Adobe DNG Converter will not necessarily
maintain all of the private metadata in certain camera-specific raw
formats because this information
is not publicly documented and therefore not available to Adobe.
However, the Adobe DNG Converter will maintain all of the original image
data as well as all of the metadata needed for a high-quality final
conversion. Arguably, the private metadata is not really archival,
regardless of the format used, simply because it is undocumented.
Nevertheless, Adobe recommends that, when photographers use the Adobe
DNG Converter for archival purposes, they should maintain both the
resulting DNG file and the original camera-specific file."

That's fine, but I can still use it if I damned well please!!

["Followup-To:" header set to rec.photo.digital.slr-systems.]
John McWilliams wrote:

You can't, as DNG doesn't preserve 100% of the metadata.
[...]
Note the words "not necessarily maintain....", certainly leaving room
for some RAW formats that have all data captured regardless.

True. But unless it is stated explicitely that for format X all
metadata is preserved, and that is proved by experimentation ...

-Wolfgang

wrote:
On 24 Mar 2008 22:38:33 GMT, in rec.photo.digital ray
wrote:

On Mon, 24 Mar 2008 16:31:25 -0400, me wrote:

On 24 Mar 2008 16:02:42 GMT, in rec.photo.digital ray
wrote:


Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to convert
back.
You can't, as DNG doesn't preserve 100% of the metadata.
Of course I can.

Not according to Adobe in:

Introducing the Digital Negative specification: Information for
Manufacturers:
http://www.adobe.com/products/dng/pd...ufacturers.pdf

"It should be noted that the Adobe DNG Converter will not necessarily
maintain all of the private metadata in certain camera-specific raw formats
because this information
is not publicly documented and therefore not available to Adobe.

Of course it is available: it's there in the file. Adobe's statement
is disingenuous. If Adobe does not funnly understand some part of
a file, they need merely copy it, as is, into their generic format
along with the info needed to put it back in the original format.


Doug McDonald