Vuescan files and Adobe DNG converter

[FrittFallFoto]

I'm using a Nikon LS-5000 filmscanner with VueScan to digitize my slides collection. This works great, and so far 5000 out of 30000 is done. I scan at 4000 dpi, 16 bits per channel, producing 115 Mb raw DNG files. Using a custom color profile, I'm able to get good results through Lightroom.

The only little annoying thing, has been that VueScan only include a very small JPEG preview, which is a problem when using fast organizing software like FastStone. So I searched the topic, and found that the DNG format allows 3 preview sizes. Furthermore I found Adobe DNG converter, allowing me to change the preview size in my existing DNG files.

OK, so I did a test with a few files, and it works. But, the filesize is reduced from 115 Mb to a variation between 72 and 85 Mb. Well, the original is uncompressed then, and the new file has some kind of lossless compression (LZW, maybe?). But if I am to convert 5000 files, I need to be a 100% sure I'm not losing any quality. So I installed RawDigger, an analyzing tool. And there seems to be no difference between the uncompressed and the compressed file, apart from the compression.

But here's the snag: In the EXIF-info extracted in RawDigger, the original file states as expected: "Compression: uncompressed". But the new file: "Compression: JPEG". This looks wrong to me, I'm 99% sure this can't be correct. But as I said, I need to 100% sure before converting 5000 files. Anyone knows for sure which compression method the Adobe DNG converter is using?

 

[tbhv55]

Anyone knows for sure which compression method the Adobe DNG converter is using?

I don't know, but here on Photoshop Help it states : "Uncompressed: No compression is applied to the raw image data." Whether or not that is strictly true, is not clear from this article.

Welcome to the forum, BTW.🙂

 

[FrittFallFoto]

Thanks for the reply. Good to be here. 🙂

I have done some more testing. I imported both the original and the new DNG file to Lightroom, did not change any settings for either of them, and exported 16-bit uncompressed TIFF files from both. The end result filesize is exactly the same, to the byte. I tried the same with JPEG files, with the same result. This tells me that the new 75 Mb DNG file contains exactly the same information as the original 115 Mb file. So there has to be some lossless compression algorithm in use in the DNG converter. Which is really good, as it saves the cost of backup disks, and enables a faster workflow. I will await a few days before converting my files, in case anyone here has some more thoughts on the subject.

 

[brbo]

I have done some more testing. I imported both the original and the new DNG file to Lightroom, did not change any settings for either of them, and exported 16-bit uncompressed TIFF files from both. The end result filesize is exactly the same, to the byte.

Which is really not that surprising. Uncompressed TIFF file of certain (pixel) size will always be the same in (byte) size, no matter of the contents.

AFAIK, Adobe tells us that converting to DNG will not add lossy compressions to the file. I don't have a reason not to believe them.

 

[jarski]

cannot comment about OP converting whole library of files. perhaps backup them before starting to work with them? before importing library to LR, you can decide what resolution preview you prefer, and it can also be saved into DNG directly from LR, no need to use DNG Converter in the middle.

I've noticed similar behaviour, although in very different environment: Adobe SW can shrink the size of original DNG after applying any changes into it. in my case original DNG was 15mb raw file from 8mp smartphone camera. after Adobe saved its changes to file, it shrunk to about 7-8mb which is closer to what I'd expect from this resolution.

 

[FrittFallFoto]

Sorry, my brain isn't functioning correctly all the time. Of course you're right, brbo. The uncompressed file will be the same if the pixel-size is. The JPEGs, however, will behave differently if the content is different. I also did a third test, this time with zip compression to the TIFF output. Same result, same filesize to the byte. This imply that the two DNG files are identical, and I believe the conclusion is that the DNG converter does what it's supposed to. Quite lovely, actually. 🙂

 

[Larry H-L]

You did not state your computer type, but my iMac, (or Photoshop) does this, it somehow "over estimates" the file sizes of photos.

 

[Particular]

I've been using the same workflow as the OP for some years now. I actually start by exporting a full 64 bit RGBI (including IR channel) vuescan raw as a tiff. I then re-import and batch export them as DNGs. I also tend to down sample to 3000 dpi unless I think the image is some hot stuff. I then batch process them in DNG converter for the compression, and import into LR.

I remember reading somewhere in the Adobe forums that the the original JPEG specification included various types of compression. The commonly used ones are lossy, but there is also a lossless type which no one seems to bother with implementing. This is the compression used in the DNG file type. So it is correct that it is JPEG, but it is not a lossy JPEG.

 

[FrittFallFoto]

Thank you, CNNY. I was not aware of the lossless JPEG compression. The issue appears to be solved then, as the RawDigger analyzing tool also is correct. Good news.

 

[philipus]

Why use DNG in this case - is it better than TIFF? I'm only curious as I've never used DNG as the final file for my scans.

br
Philip

 

[mani]

This discussion goes into some of the conflicting issues. I'm not sure whether there is a 'correct' conclusion. I always use raw tiff as output fwiw.

 

[willie_901]

Thank you, CNNY. I was not aware of the lossless JPEG compression. The issue appears to be solved then, as the RawDigger analyzing tool also is correct. Good news.

JPEGs are always lossy. That's the entire point of a JPEG... discarding information your computer screen and.or printer can not use.

An entirely different situation involves lossless JPEG editing using the jpegtran method. Software that supports jpgtran permits multiple editing of a single JPEG without cumulative. information loss.

JPEG 2000 is a relatively new, rarely used rendering standard. JPEG2000 does support lossless compression.

 

[willie_901]

DNG and TIFF are equivalent. Both can be compressed using lossless methods. Both could be compressed using lossy compression. DNG files support storing supplementary information such as all rendering parameters applied to the demosaiced, flat TIFF data.

If one know what one is doing, there is no disadvantage to using DNGs. I personally save and backup my original raws and convert copies into lossless DNGs. Others eschew DNGs for reasons I choose not to discuss.

Discussions of the advantages of DNG vs raw vs TIFF usually degrade quickly into cesspools of misinformation, misconceptions, distrust and general bad craziness.

 

[Particular]

JPEGs are always lossy.

There is technical description of how the original lossless jpeg mode is incorporated into DNG here:
http://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html

 

[jarski]

There is technical description of how the original lossless jpeg mode is incorporated into DNG here:
http://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html

seems that its technically possible. but most software products don't support this mode. besides, jpeg is just preview for the dng. it can be written and re-written into it on demand. looks like that guy has found justification of using it with his converted magic lantern files, but seems quite exotic situation that most people probably never face.

 

[willie_901]

There is technical description of how the original lossless jpeg mode is incorporated into DNG here:
http://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html

JPEG rendering from raw is lossy.

After the raw is rendered to JPEG, post-production software that uses the jpegtran plug-in can be edited over and over again (which is not rendering from raw) using lossless compression.

 

[Particular]

seems that its technically possible. but most software products don't support this mode. besides, jpeg is just preview for the dng. it can be written and re-written into it on demand. looks like that guy has found justification of using it with his converted magic lantern files, but seems quite exotic situation that most people probably never face.

The preview is just a regular lossy JPEG that is tacked onto the file. The lossless JPEG mode (which is rarely implemented in other software) is applied to the raw tiff data in the DNG. He just happens to be applying it to cinema files, but I think it is the same for still images.

 

[lawrence]

I personally save and backup my original raws and convert copies into lossless DNGs.

This is what I do for important projects, so I always maintain a copy of the original raws in a zip archive, but for general use I have the DNGs. For less important projects I simply convert to DNG and delete the original RAWs.

 

[FrittFallFoto]

I choose to trust in the lossless JPEG compression described by CNNY and jarski. I have tested more photos like before, and there is not a single piece of information lost in the new DNG's as created by Adobe DNG converter.