Vuescan File Formats: "RAW DNG" vs. "TIFF DNG"

[mafoofan]

Okay, so I think I understand the difference, but it seems like there is a lot of confusion on the internet about it and I thought it would be worth getting some confirmation here.

My understanding is that there are three ways to save a "raw" output file using Vuescan: (1) an unadjusted TIFF (non-DNG), (2) an unadjusted TIFF in a DNG wrapper ("TIFF DNG), and (3) as a "RAW" file in a DNG wrapper ("RAW DNG"). What Vuescan calls a "RAW" file is not the same as what comes out of a digital camera and should not be treated that way. It is merely an untagged record of RGB values as first recorded by Vuescan before permanent adjustments have been made.

I would have thought the "RAW DNG" is the purest, most unaltered output, and therefore the best for those who want to save all adjustments for later processing in Lightroom, Aperture, Photoshop, etc. However, apparently, this is not the case. Because the file is not tagged, ACR and other RAW converters may perform default adjustments "under the hood" the moment it is opened, unbeknownst to the user. From my understanding, these include changing the color space and increasing the gamma. I can confirm through my own comparison test that "RAW DNG" files out of Vuescan open up brighter in Lightroom than the same files that have first been converted to "TIFF DNG."

When Lightroom or ACR in Photoshop opens up a "TIFF DNG" from Vuescan, it recognizes that the image embedded within the DNG wrapper of the file is in fact an already-processed TIFF. Thus, no automatic changes are incurred.

So, it turns out, if you want the most "raw" output file from Vuescan for further editing in other software, you should--very unintuitively--avoid "RAW DNG" and choose the "TIFF" or "TIFF DNG" formats. Saving in either of those latter two formats locks-in all the processing done by Vuescan.

Then why the existence of "RAW" (again, unintuitively, saved as a TIFF unless "RAW DNG" is selected) and "RAW DNG" in Vuescan? If I am not mistaken, they are still the most "raw" files relative to Vuescan itself. Meaning, you can still re-"scan" such files in Vuescan to affect color balance, infrared-based corrections (if 64 bit RGBi has been selected), cropping, etc.

Does this all make sense, and am I right?

 

[Tim Gray]

RAW files from cameras are typically data before the demosaicing. If you look at a RAW file with no processing, it's got a checkerboard pattern to it and it's monochrome.

RAW files from scanners usually don't come from sensors with Bayer filters, so there's no RAW 'developing' (demosaicing) to be done. End of story. How they differ from TIFF output is they generally have a gamma of 1, and may or may not be inverted.

DNG is just a file format. You can put RAW data in it, before demosaicing. You can put RGB (demosaiced) data in them too. This is what linear DNGs are. Though it's just a file format, it doesn't mean that programs treat it the same as they would treat a TIFF file.

Vuescan will save your data as you wrote in many ways. TIFF, with Vuescan adjustments baked in and a gamma of ~2.2. TIFF DNG, with Vuescan adjustments baked in and I'm pretty sure a gamma of ~2.2. Or RAW DNG, no adjustments, gamma 1. Lastly, and I think the best, RAW TIFF: gamma 1, no funny file formats. All of the above are RGB data. The gamma 1 files aren't just untagged; they are gamma 1.

Programs like LR take DNG files with RGB data and a gamma of 1 and probably boost the gamma to ~2.2. Vuescan may or may not use that same ~2.2 when making the "TIFF DNG", but it does boost the gamma to something, which probably accounts for your brightness difference.

Frankly I think trying to cram in scanner files, gamma 1 or not, into LR and the like is a waste of time. Photoshop has all the tools you need to deal with a RAW gamma 1 scan file; LR doesn't necessarily. If you really need to edit your files in LR or ACR, they both accept TIFF files, so DNG doesn't offer a real advantage there. When I work with raw scan files, which is how I prefer to scan my color negs, the three most important things for me are 1) converting gamma 1 -> 2.2 properly, 2) setting the RGB black points properly, and 3) setting the RGB white points properly (less important than 2 in many cases). Setting white balance in the mid tones is a snap after that, as are regular contrast corrections, etc. By doing this stuff in LR, you trust #1 to the software (who knows what it's doing and giving up flexibility) and don't necessarily have the tools to do #2 and #3 properly.

The most 'raw' file format out of Vuescan is the RAW tiff file. Gamma 1, RGB data off the sensor, no color space attached. While the DNG RAW is nominally similar, programs don't necessarily treat DNG files the same as TIFF files and might/probably do some adjustments before displaying data (changing gamma, etc.). DNG wasn't really intended to be used for scanner data. TIFF is what you probably want; it's not unintuitive to store multichannel data, RGB or RGBI in this case, with an arbitrary gamma, usually 1, 1.8, or 2.2, in a TIFF file - that's what the format has been designed to do.

 

[mafoofan]

The bottom-line is that the "RAW TIFF" and "RAW DNG" files may be similar within Vuescan, but are not treated the same by other programs. Vuescan will treat either file as having a gamma of 1, but if you open a "RAW DNG" file from Vuescan in Lightroom or Photoshop, the gamma will automatically be boosted to 2.2. If you open a "RAW TIFF" from the exact same scan in Lightroom or Photoshop, the gamma will remain at 1.

Putting a DNG wrapper around the "raw" TIFF files is useful because it allows metadata and adjustments to be saved to the file without corrupting the underlying TIFF or requiring a secondary file to carry the information.

 

[Particular]

It sounds like you are a little confused, which is not surprising given the amount of misinformation available on the subject.

To start with, the difference between a digital camera raw file and a Vuescan raw file is that scanner files are 'linear raw files' while camera files are mosaiced (Sigma cameras being the exception here). A scanner file is basically an RGB TIFF file (three color channels) while a camera file is a single monochrome channel with proprietary (or standard if it is a DNG) de-mosaicing information tagged onto it. Hence the difference in file size between one and three channels.

In all other respects they are very similar. A raw file is a TIFF file with no, or minimal adjustments applied to it. A raw file will have no color profile, and be gamma 1.0. This is why it is so dark. When you import a raw file into ACR or Lightroom, the first thing it does is apply a generic gamma base curve to it to lighten it up (making it 1.8 or 2.2 i believe). So when you open it up and set all the sliders to 0, it is not really 0, because it assumes that you want to increase the gamma. When you export a regular TIFF (non-raw) from Vuescan it will apply a gamma curve and a color profile. If you would import this file into Lightroom, it will appear very high contrast because it is getting a second base curve applied to it.

A linear DNG file is a TIFF file with other EXIF information tagged onto it. The raw files that Vuescan exports is identical regardless of whether the ending is DNG or TIFF. You can also reprocess them in in Vuescan without the format making any difference. I think it is important to note that raw does not necessarily mean that nor processing has been applied to a file. My workflow is that I typically scan and output a 64bit RGBI TIFF, which I reprocess in order to apply the dust filter, and then I output a 48 bit DNG file for Lightroom. I also let Vuescan invert negatives which I still export as raw. Hens the output from 'Scan' or 'Save', and the 'Raw save film' option (confusingly named unfortunately)

The logic to outputting a DNG as opposed to a TIFF for use in Lightroom is firstly just the convenience of not having the sidecar files in Lightroom (the DNG format embeds the settings in the file). The most important difference is in the Camera Calibration pane in Lightroom. When you import a camera raw file, Lightroom recognizes the camera and offers color profiles specific to your camera sensor. When you import a generic TIFF, it will apply default settings that you have no control over. With a DNG file from Lightroom it will recognize your scanner as a 'camera'. Using the DNG Profile Editor (free from adobe) you will be able to make custom scan profiles for your scanner and apply them to different film types. For instance I have profiles that have a base curve that I use with negative scans, and ones without base curve for scanned slides because slides are 'gamma correct' already.
It is this functionality that makes using DNG really powerful. I believe that Apple Aperture only accepts camera DNG's but not linear DNG's (I'm not certain if this is still true) so if you use that, it is best to output a raw TIFF.

 

[Tim Gray]

ACR saves adjustments in TIFF and JPEG files too. Not sure about LR. Regardless, storing those adjustments in the metadata is useless to me if I can't make all the adjustments I need to make in said program. And last I checked, ACR and LR didn't allow for RGB curves. That could have changed in the last couple months; I don't know.

LR boosts the gamma of RAW camera files too. Which is why I wouldn't store my linear scanner data in a DNG file. If I have a scan file with linear gamma, I want my image editing program to open it up as linear gamma - I chose linear gamma because I clearly want to do something with the gamma down the road, not have some program do it automatically for me.

You make this distinction that Vuescan is treating your raw DNG files differently then other programs treat DNG files. Maybe this is because Vuescan is basically the only program I can think of that thinks DNG files might be anything other than from a digital camera, i.e. scanner data. That's basically my argument why I wouldn't use it for scans - every other program I can think of treats a DNG file like it came from a digital camera. No such assumptions are made for TIFF files.

 

[mafoofan]

I think we are talking past each other to some extent.

I understand there is a difference between camera-derived and scanner-derived "RAW" files. I also understand what DNG is. I tried to make both points clear in my original post.

What I find confusing about Vuescan is that it offers multiple output formats that could be called "RAW": (1) straight, unadjusted TIFF with no DNG wrapper, (2) the same TIFF, but DNG wrapped (a "TIFF DNG" in Vuescan parlance), (3) a "RAW" output (saved as a TIFF), and (4) the same "RAW" file saved with a DNG wrapper ("RAW DNG"). It is important for the user to know the differences so he can properly integrate the appropriate format into his workflow. Whether one wants to use a DNG wrapper is really a separate discussion--what's key here is that what Vuescan calls "RAW" is not necessarily "RAW" the way many of us might take it to mean.

Visually, it is obvious that Lightroom treats "RAW DNG" and "TIFF DNG" files differently. The former is given an obvious gamma boost while the latter appears to remain unchanged. If you are telling me some adjustments to gamma levels and curves are being applied to the "TIFF DNG" out of Vuescan, then I would love to know how to check what these changes are and learn to undo them so I can retain full manual control over my files.

 

[Particular]

LR boosts the gamma of RAW camera files too. Which is why I wouldn't store my linear scanner data in a DNG file. If I have a scan file with linear gamma, I want my image editing program to open it up as linear gamma - I chose linear gamma because I clearly want to do something with the gamma down the road, not have some program do it automatically for me.

The linear scanner data in a DNG file is identical to that data in a TIFF file. LR does not alter this data in any way, it only applies the gamma curve when it displays it and when it processes it for export. To have control over what gamma is being applied you would need to make custom profiles which is fairly easy to do. I think the choice between Lightroom or ACR+Photoshop or Photoshop (or any other program), has more to do with personal work flow preferences. In my case I prefer Lightroom as I can manage digital files together with scanned files, and only take a file to Photoshop if I run into limitations.
For Lightroom DNG is the best format, but for Photoshop you are probably better off using TIFF.

 

[Tim Gray]

I realize what you are saying. And I'm saying this:

1) Vuescan Raw files aren't 'RAW' as most people understand them, since most people understand them to be camera derived files. Which you get since that was your 2nd sentence in your most recent reply.

2) The 'rawest' file you'll get out of Vuescan is to select 'Raw' no DNG. It's linear gamma and programs don't make the assumption as to what final gamma you want. That is NOT true if you pick DNG.

The other bit about Vuescan (forgot about this) is that regardless if you chose DNG or not, if you choose 'Raw', you can have Vuescan save the data before or after certain corrections. See the 'Raw output with' and 'Raw save film' options. If you really want the 'rawest' version of your data, set the first to 'scan' and uncheck the second.

http://www.hamrick.com/vuescan/html/vuesc32.htm#outputrawfile

 

[mafoofan]

The linear scanner data in a DNG file is identical to that data in a TIFF file. LR does not alter this data in any way, it only applies the gamma curve when it displays it and when it processes it for export. To have control over what gamma is being applied you would need to make custom profiles which is fairly easy to do. I think the choice between Lightroom or ACR+Photoshop or Photoshop (or any other program), has more to do with personal work flow preferences. In my case I prefer Lightroom as I can manage digital files together with scanned files, and only take a file to Photoshop if I run into limitations.
For Lightroom DNG is the best format, but for Photoshop you are probably better off using TIFF.

The point I've been trying to make is that this represents a false dichotomy. In Vuescan, the choice is not between "DNG" and "TIFF." You can have a non-DNG TIFF, a TIFF with a DNG wrapper, what Vuescan calls a "RAW" file saved as a TIFF, or the same "RAW" file with a DNG wrapper.

There are two different sorts of DNG possible with Vuescan, and two different sorts of TIFF.

 

[Tim Gray]

The linear scanner data in a DNG file is identical to that data in a TIFF file. LR does not alter this data in any way, it only applies the gamma curve when it displays it and when it processes it for export. To have control over what gamma is being applied you would need to make custom profiles which is fairly easy to do.

That is true. However, I find it a lot easier to adjust gamma with the levels or curves dialogue in PS than adjusting profiles in LR.

The point I've made in the past about LR not being great for scans (granted, I seem to be the only person in the world who thinks this) is that as far as I know, it doesn't provide the right tools for me to properly color correct scan files. I find a full featured curves tool indispensable. I realize I'm in the minority here, but then again, I see an awful lot of scans with purple shadows from people who do their scans in LR.

 

[mafoofan]

1) Vuescan Raw files aren't 'RAW' as most people understand them, since most people understand them to be camera derived files. Which you get since that was your 2nd sentence in your most recent reply.

Didn't I also say this in my original post?

"What Vuescan calls a 'RAW' file is not the same as what comes out of a digital camera and should not be treated that way."

2) The 'rawest' file you'll get out of Vuescan is to select 'Raw' no DNG. It's linear gamma and programs don't make the assumption as to what final gamma you want. That is NOT true if you pick DNG.

So, you are saying that if one were to check no boxes but the RAW box, he would wind up with a TIFF file that other programs such as Lightroom will not attempt to apply default adjustments to? I haven't tried that, honestly.

And you are further saying that if one were to save as a "TIFF DNG" (not a "RAW DNG"), he would be stuck with a file that Lightroom and Photoshop apply gamma adjustments to?

The other bit about Vuescan (forgot about this) is that regardless if you chose DNG or not, if you choose 'Raw', you can have Vuescan save the data before or after certain corrections. See the 'Raw output with' and 'Raw save film' options. If you really want the 'rawest' version of your data, set the first to 'scan' and uncheck the second.

Yes, this is all true, but does not help clarify between the four output types in question.

 

[Tim Gray]

The point I've been trying to make is that this represents a false dichotomy. In Vuescan, the choice is not between "DNG" and "TIFF." You can have a non-DNG TIFF, a TIFF with a DNG wrapper, what Vuescan calls a "RAW" file saved as a TIFF, or the same "RAW" file with a DNG wrapper.

There are two different sorts of DNG possible with Vuescan, and two different sorts TIFF.

1) TIFF - normal RGB data, corrected gamma, all other Vuescan post processing baked into the file
2) TIFF/DNG - same as above as a DNG
3) Raw - normal RGB data, gamma 1 (most of the time), IR cleaning and film curve optional
4) Raw/DNG - same as above as a DNG.

Select 1 or 3 if you want to use your files in most image editors. Select 2 or 4 if you want to use your DNG in LR or ACR.

Select 1 or 2 if you don't want to mess with linear gamma files and are ok with the basic conversion Vuescan does (inverting, film curves, etc.). Select 3 or 4 if you want a linear gamma file and want to do everything.

I did find out that if you save a scan as an 8 bit Raw, it is gamma 2.2. Otherwise it is gamma 1.

 

[Tim Gray]

So, you are saying that if one were to check no boxes but the RAW box, he would wind up with a TIFF file that other programs such as Lightroom will not attempt to apply default adjustments to? I haven't tried that, honestly.

Pretty sure. There are a lot of boxes in Vuescan to check 🙂 but that's how I scan my files. Raw TIFFS. They open up as linear files in PS and ACR. I don't have LR on me at the moment, but I would imagine they would be the same there.

And you are further saying that if one were to save as a "TIFF DNG" (not a "RAW DNG"), he would be stuck with a file that Lightroom and Photoshop apply gamma adjustments to?

I haven't tried it. Your TIFF DNG file should have a gamma of 2.2 encoded, so I'd wager that LR would recognize that and not adjust it to 2.2. I'm also guessing that LR automatically boosts gamma to 2.2 for any DNG files that are flagged as gamma 1.

 

[mafoofan]

Tim, if I save as a TIFF or TIFF DNG (#1 and #2), but apply no color balancing and no other adjustments, the output should be identical compared to #3 and #4, from the point of view of Vuescan, correct?

What I want is a file I can work on in Lightroom that is as unmolested as possible, without the need for sidecar files.

 

[mafoofan]

I haven't tried it. Your TIFF DNG file should have a gamma of 2.2 encoded, so I'd wager that LR would recognize that and not adjust it to 2.2. I'm also guessing that LR automatically boosts gamma to 2.2 for any DNG files that are flagged as gamma 1.

Hmm. When I open a "RAW DNG" file in Lightroom, it is much lighter than the equivalent, unadjusted "TIFF DNG." This accords with what I had read elsewhere: that saving as a 16/48-bit TIFF in Vuescan will leave you with a file that has a gamma value of 1, and this--somehow--prevents programs like Lightroom from boosting the gamma on its own. In contrast, the "RAW" files outputted by Vuescan get gamma boosted because ACR and Lightroom don't know what to do with the data and resort to default adjustments.

 

[Particular]

I think we are talking past each other to some extent.

I understand there is a difference between camera-derived and scanner-derived "RAW" files. I also understand what DNG is. I tried to make both points clear in my original post.

What I find confusing about Vuescan is that it offers multiple output formats that could be called "RAW": (1) straight, unadjusted TIFF with no DNG wrapper, (2) the same TIFF, but DNG wrapped (a "TIFF DNG" in Vuescan parlance), (3) a "RAW" output (saved as a TIFF), and (4) the same "RAW" file saved with a DNG wrapper ("RAW DNG"). It is important for the user to know the differences so he can properly integrate the appropriate format into his workflow. Whether one wants to use a DNG wrapper is really a separate discussion--what's key here is that what Vuescan calls "RAW" is not necessarily "RAW" the way many of us might take it to mean.

Visually, it is obvious that Lightroom treats "RAW DNG" and "TIFF DNG" files differently. The former is given an obvious gamma boost while the latter appears to remain unchanged. If you are telling me some adjustments to gamma levels and curves are being applied to the "TIFF DNG" out of Vuescan, then I would love to know how to check what these changes are and learn to undo them so I can retain full manual control over my files.

Vuescan IS confusing. I find the naming of some settings really convoluted.
Basically which of the 4 options you choose depends on your workflow.
Firstly you need to ask yourself if you need a DNG or if a TIFF is better (see discussion above).
Secondly you have the choice to let Vuescan do some of the preprocessing, like inverting a negative or applying a dust filter, or just output a totally unadulterated file and that you do all the processing in another program. The great thing about the raw output option is that you can do some processing and still have Lightroom treat it as if it is truly raw.

I was under the impression that LR applied a gamma curve to both DNG and TIFF files by default, but if the raw TIFFs that you import are really dark, that may not be the case (are you using LR4, perhaps they changed something?).
I believe the gamma curve is the only default adjustment, so you can easily override it with the curves adjustment in LR. If the contrast is too high, you could apply a 'counter curve' to bring the contrast down. This is of course rather convoluted, but there is no penalty in dong this, because in LR (and ACR) the final output curve is the sum of the base curve and your personal settings. In the end it only applies a curve once.

Either way, if you want to understand what is happening under the hood read some of the DNG Profile Editor documentation:
http://labs.adobe.com/wiki/index.php/DNG_Profiles
You may even want to make your own profiles, but it helps just to understand it better.

 

[Tim Gray]

I'm no expert on LR. However, if you can't think of any reason to have linear gamma files, I'd say avoid linear gamma files and just go with the TIFF DNG. Two reasons for linear gamma files are 1) you use some program like ColorPerfect which requires them or 2) you are like me and prefer to do your own conversions in PS. Reasons to avoid linear gamma files are that they can be a pain, as you are finding out 🙂

Actually, now that I think about it, I'm not sure if Vuescan can save TIFF files (non-Raw files that is) without applying the film curves; I can't recall. Even the 'generic film' type makes some assumptions, though nothing too serious.

Tim, if I save as a TIFF or TIFF DNG (#1 and #2), but apply no color balancing and no other adjustments, the output should be identical compared to #3 and #4, from the point of view of Vuescan, correct?

I would think no. #3 and #4 are linear gamma files, #1 and #2 aren't. If you were to load them up again in Vuescan, Vuescan would apply the same gamma curves to #3/4 for display and saving that it did in generating #1/2. That gamma curve is not necessarily the same thing that LR applies - I really don't know what curve Vuescan or LR apply - it's nominally around 2.2, but I've never seen what the actual curves are for either program.

 

[Particular]

That is true. However, I find it a lot easier to adjust gamma with the levels or curves dialogue in PS than adjusting profiles in LR.

The point I've made in the past about LR not being great for scans (granted, I seem to be the only person in the world who thinks this) is that as far as I know, it doesn't provide the right tools for me to properly color correct scan files. I find a full featured curves tool indispensable. I realize I'm in the minority here, but then again, I see an awful lot of scans with purple shadows from people who do their scans in LR.

I've seen those 'purple shadow scans' too. Lightroom was clearly designed with digital camera files in mind, not scans. The best way to get good results in LR is to feed it files that are more like camera files. Using custom profiles is one way of achieving that. When the colors are 'off' in an image it can be tricky (but not impossible) to get them right in LR. My reason for wanting to use Lightroom over Photoshop has really more to do with the ease of managing and applying basic adjustments to large batches of files. I revert to Photoshop only when I need total control.

 

[mafoofan]

Thanks for all the discussion everyone. After reading through everything, I think I understand a little better. However, I've been experimenting more with the "RAW" files out of Vuescan, and don't know what to make of what's happening now.

If you open up RAW TIFFs in Lightroom, you get what you'd expect--a very dark, unboosted gamma image. If you open up RAW DNGs in Lightroom, you also get what you'd expect given the information discussed above: an image that has been gamma-boosted, but which is otherwise the same as the RAW TIFF.

If you "re-scan" the RAW DNG in Vuescan and output a RAW TIFF, the re-scanned RAW TIFF looks just like the other, original RAW TIFF in Lightroom: dark with an unadjusted, scrunched up histogram.

Okay, so far so good.

However, here is where things get tricky--and I'm not sure whether the problem is with Lightroom or Vuescan. If I convert the RAW TIFF to a DNG file in Lightroom (checking the option to embed the original raw file), two issues arise: (1) the file size drops from ~145mb to ~95mb, and (2) Vuescan is no longer able to extract the original RAW TIFF from the new DNG.

Similarly, if I save Lightroom adjustments to a RAW DNG file from Vuescan, the file size decreases and Vuescan can no longer extract the original RAW TIFF.

In short, DNG is not working the way I understand it should, as the original, unadjusted image file is not being preserved after Lightroom has made adjustments.

Thoughts? Am I being an idiot?

 

[Particular]

There is no reason to embed the original file in a DNG. The option exists I believe for legal reasons in law enforcement in order to ensure that photographic evidence has not been altered. The image data in the TIFF and in the DNG is identical, so you would just be doubling up. If you save the Lightroom adjustments in a DNG file, it is saving the position of the sliders, so you can for instance open them on another computer with the same settings. The idea of using raw is that the raw data remains unaltered, and you can always revert to it, by moving all the sliders to 0.
The reduction in file size is because DNG has a lossless compression option which can cut the file size by about 1/3. I in fact output DNGs from Vuescan and then re-encode them with the DNG converter which applies a newer version of DNG than Vuescan in order to get an even smaller file. It is one of the things I like about DNG. With a lot of scans it saves a significant amount of HD space.