'extra' lossless compression for camera raw images

"KandorMan" wrote in message
...
On 19 Mar 2008 16:47:38 GMT, ray wrote:

On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:

I just finished writing a small tool (sgraw) which can losslessly
compress camera raw images, usually between 20-to-60%. (Regulars here
might recall I have mentioned this 'possibility' earlier, I finally put
in some hours to actually build a working demo :-)

http://www.sachingarg.com/compression/sgraw/

That would be savings between 1GB to 3GB for every 5GB of data.

Even when camera has already compressed the raw image (which isn't
always the case), its still possible to squeeze out extra compression as
cameras have to use simple algorithms for speed.

Compression is totally lossless, we get back the exact file we start
with (just like with zip), so both pixel information and meta-data are
perfectly restored.

This is just the first quick hack version (and supports all nikon and
fuji cameras), so it obviously wouldn't make sense to actually use it
for backups etc, but if there is enough interest I can build it further
to support other raw formats, along with improved compression and speed.

Your thoughts on this?

It seems totally superfluous since there are already dozens of
compression programs available. What makes you think you've discovered
some 'new and improved' algorithm?

Agreed. I just used WinRAR on a couple of RAW files and got anywhere from
38% to
52% compression rates by just using its auto defaults. RAR is a well-known
and
universal compression method that works well for all multimedia formats.

Yes, general purpose compressors like rar/bzip2/zip etc can give good
results if original raw format did not use any compression (lossless). sgraw
is tuned for raw images, so it can give consistently better results than
them (theoritically at better speed too, but current implementation is slow.
I will try to improve this).

Besides, it can further compress the raw files even if they were already
compressed. Compression ratio is relatively less in this case as there are
technical limitations.

I would love to see any results different from what I found in my testing.
Which camera's raw files did you use for your tests? If these were from a
Nikon or Fuji, I am hoping that you will be amazed to see the difference
when using sgraw. If it was some other camera then we will have to wait till
I can work on those formats.

Unless
you can consistently do greater than 50-60% on all RAW files it's not
worth
anyone's time to test this for you. Well, except as your own as a learning
exercise and a way to get attention for wasting everyone else's time and
resources in doing so.

Fortunately, not everyone uses 'all' the cameras :-) If its getting results
on your raw files which are good enough for you, I will be happy to know
that you can find it useful. Otherwise, my apologies for wasting your time.
(If rar is getting those results you mentioned on your raw files, I am very
optimistic that you will like what sgraw will manage to do for you :-)

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

On Mon, 24 Mar 2008 10:13:05 +0530, Sachin Garg wrote:

"KandorMan" wrote in message
...
On 19 Mar 2008 16:47:38 GMT, ray wrote:

On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:

I just finished writing a small tool (sgraw) which can losslessly
compress camera raw images, usually between 20-to-60%. (Regulars here
might recall I have mentioned this 'possibility' earlier, I finally
put in some hours to actually build a working demo :-)

http://www.sachingarg.com/compression/sgraw/

That would be savings between 1GB to 3GB for every 5GB of data.

Even when camera has already compressed the raw image (which isn't
always the case), its still possible to squeeze out extra compression
as cameras have to use simple algorithms for speed.

Compression is totally lossless, we get back the exact file we start
with (just like with zip), so both pixel information and meta-data
are perfectly restored.

This is just the first quick hack version (and supports all nikon and
fuji cameras), so it obviously wouldn't make sense to actually use it
for backups etc, but if there is enough interest I can build it
further to support other raw formats, along with improved compression
and speed.

Your thoughts on this?

It seems totally superfluous since there are already dozens of
compression programs available. What makes you think you've discovered
some 'new and improved' algorithm?

Agreed. I just used WinRAR on a couple of RAW files and got anywhere
from 38% to
52% compression rates by just using its auto defaults. RAR is a
well-known and
universal compression method that works well for all multimedia
formats.

Yes, general purpose compressors like rar/bzip2/zip etc can give good
results if original raw format did not use any compression (lossless).
sgraw is tuned for raw images, so it can give consistently better
results than them (theoritically at better speed too, but current
implementation is slow. I will try to improve this).

Besides, it can further compress the raw files even if they were already
compressed. Compression ratio is relatively less in this case as there
are technical limitations.

I would love to see any results different from what I found in my
testing. Which camera's raw files did you use for your tests? If these
were from a Nikon or Fuji, I am hoping that you will be amazed to see
the difference when using sgraw. If it was some other camera then we
will have to wait till I can work on those formats.

Unless
you can consistently do greater than 50-60% on all RAW files it's not
worth
anyone's time to test this for you. Well, except as your own as a
learning exercise and a way to get attention for wasting everyone
else's time and resources in doing so.

Fortunately, not everyone uses 'all' the cameras :-) If its getting
results on your raw files which are good enough for you, I will be happy
to know that you can find it useful. Otherwise, my apologies for wasting
your time. (If rar is getting those results you mentioned on your raw
files, I am very optimistic that you will like what sgraw will manage to
do for you :-)

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to convert
back.

On Mon, 24 Mar 2008 16:31:25 -0400, me wrote:

On 24 Mar 2008 16:02:42 GMT, in rec.photo.digital ray
wrote:


Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to convert
back.

You can't, as DNG doesn't preserve 100% of the metadata.

Of course I can.

wrote:
On 24 Mar 2008 22:38:33 GMT, in rec.photo.digital ray
wrote:

On Mon, 24 Mar 2008 16:31:25 -0400, me wrote:

On 24 Mar 2008 16:02:42 GMT, in rec.photo.digital ray
wrote:


Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to convert
back.
You can't, as DNG doesn't preserve 100% of the metadata.
Of course I can.

Not according to Adobe in:

Introducing the Digital Negative specification: Information for
Manufacturers:
http://www.adobe.com/products/dng/pd...ufacturers.pdf

"It should be noted that the Adobe DNG Converter will not necessarily
maintain all of the private metadata in certain camera-specific raw formats
because this information
is not publicly documented and therefore not available to Adobe. However,
the Adobe DNG Converter will maintain all of the original image data as
well as all of the metadata needed for a high-quality final conversion.
Arguably, the private metadata is not really archival, regardless of the
format used, simply because it is undocumented. Nevertheless, Adobe
recommends that, when photographers use the Adobe DNG Converter for
archival purposes, they should maintain both the resulting DNG file and the
original camera-specific file."

Note the words "not necessarily maintain....", certainly leaving room
for some RAW formats that have all data captured regardless.

--
John McWilliams

On Mon, 24 Mar 2008 19:15:29 -0400, me wrote:

On 24 Mar 2008 22:38:33 GMT, in rec.photo.digital ray
wrote:

On Mon, 24 Mar 2008 16:31:25 -0400, me wrote:

On 24 Mar 2008 16:02:42 GMT, in rec.photo.digital ray
wrote:


Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to
convert back.

You can't, as DNG doesn't preserve 100% of the metadata.

Of course I can.

Not according to Adobe in:

Introducing the Digital Negative specification: Information for
Manufacturers:
http://www.adobe.com/products/dng/pd...ufacturers.pdf

"It should be noted that the Adobe DNG Converter will not necessarily
maintain all of the private metadata in certain camera-specific raw
formats because this information
is not publicly documented and therefore not available to Adobe.
However, the Adobe DNG Converter will maintain all of the original image
data as well as all of the metadata needed for a high-quality final
conversion. Arguably, the private metadata is not really archival,
regardless of the format used, simply because it is undocumented.
Nevertheless, Adobe recommends that, when photographers use the Adobe
DNG Converter for archival purposes, they should maintain both the
resulting DNG file and the original camera-specific file."

That's fine, but I can still use it if I damned well please!!

["Followup-To:" header set to rec.photo.digital.slr-systems.]
John McWilliams wrote:

You can't, as DNG doesn't preserve 100% of the metadata.
[...]
Note the words "not necessarily maintain....", certainly leaving room
for some RAW formats that have all data captured regardless.

True. But unless it is stated explicitely that for format X all
metadata is preserved, and that is proved by experimentation ...

-Wolfgang

wrote:
On 24 Mar 2008 22:38:33 GMT, in rec.photo.digital ray
wrote:

On Mon, 24 Mar 2008 16:31:25 -0400, me wrote:

On 24 Mar 2008 16:02:42 GMT, in rec.photo.digital ray
wrote:


Actually, I have found that converting to DNG does quite well. And, of
course, DNG is supported already by photo editors, so no need to convert
back.
You can't, as DNG doesn't preserve 100% of the metadata.
Of course I can.

Not according to Adobe in:

Introducing the Digital Negative specification: Information for
Manufacturers:
http://www.adobe.com/products/dng/pd...ufacturers.pdf

"It should be noted that the Adobe DNG Converter will not necessarily
maintain all of the private metadata in certain camera-specific raw formats
because this information
is not publicly documented and therefore not available to Adobe.

Of course it is available: it's there in the file. Adobe's statement
is disingenuous. If Adobe does not funnly understand some part of
a file, they need merely copy it, as is, into their generic format
along with the info needed to put it back in the original format.


Doug McDonald