Vuescan File Formats: "RAW DNG" vs. "TIFF DNG"

[mafoofan]

There is no reason to embed the original file in a DNG. The option exists I believe for legal reasons in law enforcement in order to ensure that photographic evidence has not been altered. The image data in the TIFF and in the DNG is identical, so you would just be doubling up. If you save the Lightroom adjustments in a DNG file, it is saving the position of the sliders, so you can for instance open them on another computer with the same settings. The idea of using raw is that the raw data remains unaltered, and you can always revert to it, by moving all the sliders to 0.
The reduction in file size is because DNG has a lossless compression option which can cut the file size by about 1/3. I in fact output DNGs from Vuescan and then re-encode them with the DNG converter which applies a newer version of DNG than Vuescan in order to get an even smaller file. It is one of the things I like about DNG. With a lot of scans it saves a significant amount of HD space.

I understand what DNG is supposed to do. But in this case, the DNG files created or modified by Lightroom appear to corrupt the underlying raw data rather than preserve it. If things were functioning as expected, shouldn't I be able to pull the original, non-adjusted TIFF out of the DNG wrapper no matter what changes I've made in Lightroom?

 

[Particular]

I understand what DNG is supposed to do. But in this case, the DNG files created or modified by Lightroom appear to corrupt the underlying raw data rather than preserve it. If things were functioning as expected, shouldn't I be able to pull the original, non-adjusted TIFF out of the DNG wrapper no matter what changes I've made in Lightroom?

I just reread your post above. If you are doing the conversion from TIFF to DNG in Lightroom (or DNG converter) there are two issues.
First of all Lightroom will continue to see the files as originating as a generic TIFF and not see them as 'camera files' (where the scanner is the 'camera'). This then makes it much harder to apply DNG profiles, if this is the workflow route you would choose to follow. Only Vuescan will tag the file correctly with the scanner name. I tried to find a way to edit the exif manually to do this, but didn't find an easy way. I just always let Vuescan do the initial conversion.
Second Lightroom is encoding it in a newer version of DNG than Vuescan would (which explains the smaller file size), so it is possible that that is why Vuescan won't extract it. This is an anomaly, and a little ironic that DNG would have limited backwards compatibility, given that that is one of the arguments for using the format to start with. There is a setting to encode in earlier versions of DNG which may be the simple way around this. I believe the main difference is in the enhanced lossless compression, I haven't noticed any other improvements in the files.

There may also be a way to revert to the raw TIFF file using photoshop, but I don't think you can open DNG files directly without using ACR, which would be applying the base curve again.
If you do have embedded originals, You can extract them with the Adobe DNG converter.

 

[mafoofan]

^^^

The problem with allowing Vuescan to do the initial conversion to DNG (checking both "Raw file" and "Raw DNG format" when scanning), is as I described above. To repeat: if you open up that DNG file in Lightroom, then save adjustments made in Lightroom to the file, you can no longer extract the untouched, raw TIFF from the DNG. Beyond that, you also get stuck with the automatic gamma adjustment in Lightroom.

 

[sepiareverb]

Not to be dumb, but how does one open a TIFF with ACR? My RAW files from the cameras open with that but I can't work a TIFF scan in ACR- it is just a plug-in for the camera files no?

 

[Particular]

^^^

The problem with allowing Vuescan to do the initial conversion to DNG (checking both "Raw file" and "Raw DNG format" when scanning), is as I described above. To repeat: if you open up that DNG file in Lightroom, then save adjustments made in Lightroom to the file, you can no longer extract the untouched, raw TIFF from the DNG. Beyond that, you also get stuck with the automatic gamma adjustment in Lightroom.

That may be correct. Although logic suggest there should be a way. I may need to do some tests later.
For me however this is not a problem, because I have made profiles with all adjustments at 0, and that leave the gamma untouched. This allows me to take it into either Lightroom or Photoshop the same way as a raw TIFF.
I do this regularly with scans from slides or from prints. I then have a starting point that is very close to the original.

 

[cabbiinc]

http://www.hamrick.com/vuescan/html/vuesc19.htm#topic16

No, it's not a RAW file like you get from a camera. It's more of a 'less processed' Vuescan file. But just activating the RAW file isn't everything. There are other factors in getting as least processed of a file as can be saved.

Setting the RAW Output With to Scan, compression and size reduction off (obviously), RAW File Type to 64 bit RGBI. If you set the RAW Output to Save more processing, including IR dust removal, gets added into the mix. Setting it to Scan saves the IR channel as a separate layer and can be used or discarded later.

 

[Tim Gray]

Not to be dumb, but how does one open a TIFF with ACR? My RAW files from the cameras open with that but I can't work a TIFF scan in ACR- it is just a plug-in for the camera files no?

One of two ways. If you just want to play around with it, go to the File->Open menu. Navigate to your file. In the file format drop down menu, change it to Camera Raw.

The second way is to modify your ACR preferences. There's 3 choices if I recall correctly: Never open TIFF/JPG in ACR, Always open TIFF/JPG in ACR, Open in ACR only if ACR settings are found in the file. I think you can set it separately for JPG and TIFF as well.

 

[Particular]

Not to be dumb, but how does one open a TIFF with ACR? My RAW files from the cameras open with that but I can't work a TIFF scan in ACR- it is just a plug-in for the camera files no?

Correct, you don't open TIFFs up in ACR. If you either are scanning or you import the TIFF in Vuescan you can save it as a DNG file which will open in ACR. The raw TIFF option in Vuescan is intended either for reprocessing in Vuescan, or for opening directly in Photoshop (bypassing ACR), or another program like Aperture.

 

[Brian Puccio]

Huh, I feed LR (3, now 4) plain 16-bitt TIFF files from my Coolscan 4000 powered by Vuescan and have no issues with dark images or any other sorts of issues.

Mostly Provia, some Velvia and a handful of B&W (mostly Adox CMS 20).

 

[sepiareverb]

One of two ways. If you just want to play around with it, go to the File->Open menu. Navigate to your file. In the file format drop down menu, change it to Camera Raw.

Thank you Tim! I shall never open Lightroom again.

 

[mafoofan]

Huh, I feed LR (3, now 4) plain 16-bitt TIFF files from my Coolscan 4000 powered by Vuescan and have no issues with dark images or any other sorts of issues.

Mostly Provia, some Velvia and a handful of B&W (mostly Adox CMS 20).

Are you feeding it "raw" TIFFs or adjusted TIFFs from Vuescan? Vuescan outputs TIFFs two different ways. You either check "TIFF" and wind up with an adjusted TIFF file, which would explain why it would look close to decent in LR, or you check "Raw" and wind up with an unadjusted TIFF. The unadjusted TIFF will look dark and shadowy.

Speaking of raw TIFFs in Lightroom: I am now toying with them, but wonder what the most kosher method is to correct for the gamma of 1 inherent to the file. It looks like the most expedient way is to increase the Exposure, but I don't really know what that's doing and if it is equivalent to boosting the gamma.

 

[Brian Puccio]

Are you feeding it "raw" TIFFs or adjusted TIFFs from Vuescan? Vuescan outputs TIFFs two different ways. You either check "TIFF" and wind up with an adjusted TIFF file, which would explain why it would look close to decent in LR, or you check "Raw" and wind up with an unadjusted TIFF. The unadjusted TIFF will look dark and shadowy.

Speaking of raw TIFFs in Lightroom: I am now toying with them, but wonder what the most kosher method is to correct for the gamma of 1 inherent to the file. It looks like the most expedient way is to increase the Exposure, but I don't really know what that's doing and if it is equivalent to boosting the gamma.

From the manual:

http://www.hamrick.com/vuescan/html/vuesc32.htm#outputtab

Output | TIFF file
This enables writing the cropped and processed images to a TIFF file when scanning.

...

Output | Raw file
(Professional Edition only)

This enables writing raw data from the scanner to a TIFF file.

The raw files are the result of the first of two steps VueScan performs: "scanning". The second step is "processing". These steps are described in the topic "How VueScan Works" in this User's Guide. By providing a mechanism to cleanly separate these two steps, VueScan provides great flexibility and offers options not available in most other scanner software.

You can perform the "processing" step later by setting Input | Source to "File". This lets you rescan images with different settings, without needing to rescan the media.

When the raw file is written depends on the setting of Output | Raw output with.

Raw files are always stored as TIFF files and can therefore be examined in image viewing and editing programs. Note, however that the resolution of raw files (as specified in Output | Raw file type) may be greater than some programs can read.

Little processing is done on raw files so they are a close representation of exactly what the scanner has produced. Raw files will not be filtered nor will color settings be applied. As a result, raw scans may look "wrong".

Raw files contain as much data as the scanner was able to produce before any modifications may have been made, and are therefore good for archiving.

The image gamma value is 1.0 when there are two bytes (16-bits) per sample, and 2.2 when there is one byte (8-bits) per sample. Raw files saved with gamma 1.0 will look dark, but this is normal.

The one exception to this is if Output | Raw output with is set to "Save". In this case, the infrared cleaning and grain reduction is also done before saving the raw scan files.

Advanced Option: This option is displayed when Prefs | Enable raw from disk is set or when not scanning from a file.

I use the first option, the TIFF option.

I lock the exposure and film base color as described here: http://www.hamrick.com/vuescan/html/vuesc15.htm#topic12

And I set the color balance, as described by http://www.hamrick.com/vuescan/html/vuesc31.htm#colorcolorbalance to be either none (just gamma corrected) or neutral (gamma corrected and adjustment based upon white point and black point to stretch the image's intensity range), though I use the neutral option quite sparingly.

The TIFF from this procedure gives me a very accurate representation of the slide which I then "futz with" slightly in LR, prefering to keep things as captured for the most part. In short, sharping, maybe some horizon leveling, maybe cropping a tad, sometimes moving highlights and shadows, but otherwise leaving the image as is.

 

[mafoofan]

^^^ That's why you aren't experiencing what others are. The TIFF files you are outputting are not the "raw" TIFF files that we've been talking about.

 

[cabbiinc]

From the manual:



I use the first option, the TIFF option.

I lock the exposure and film base color as described here: http://www.hamrick.com/vuescan/html/vuesc15.htm#topic12

And I set the color balance, as described by http://www.hamrick.com/vuescan/html/vuesc31.htm#colorcolorbalance to be either none (just gamma corrected) or neutral (gamma corrected and adjustment based upon white point and black point to stretch the image's intensity range), though I use the neutral option quite sparingly.

The TIFF from this procedure gives me a very accurate representation of the slide which I then "futz with" slightly in LR, prefering to keep things as captured for the most part. In short, sharping, maybe some horizon leveling, maybe cropping a tad, sometimes moving highlights and shadows, but otherwise leaving the image as is.

You are ouputting a Tiff file rather than a Jpeg, but it's still a processed file. On the Color tab, Output Color Space, if you use the Apple RGB, ColorMatch RGB, ProPhoto RGB or ECI RGB color space, the image gamma will be 1.8. If you use any other color space, the image gamma will be 2.2. This processing is not included in the RAW files if you save the RAW file from Scan or Preview, but if you save the RAW file from Save it will, along with other things like IR cleaning and color corrections.

 

[mafoofan]

Well, I tried using ColorPerfect, but found that it doesn't really provide me with an advantage over what I'm already getting out of my workflow--it's also a more disruptive process as you can't really use Lightroom for organizational purposes anymore without great hassle.

I'm finding that locking the exposure and film base in Vuescan results in raw TIFFs that are already close to perfectly white-balanced and I only need to adjust for taste. Also, I don't like that ColorPerfect is somewhat opaque--I don't really know what adjustments it's making.

So, here's the question, if I'm working in Lightroom off of raw TIFFs from Vuescan, what is the best way to boost the gamma of the files? Simply playing with the tone curve doesn't work, as you cannot push many of the midtones and highlights bright enough. Using the "Exposure" slider does seem to work--but I don't know what it's really doing.

Is there a simple way to adjust gamma in Lightroom?

 

[Tim Gray]

Not that I know of. One way change the gamma of a file is to assign the original with a profile that has been adjusted to have gamma 1 and then convert it to the original profile. For example, make a profile based on AdobeRGB but with gamma 1 and assign that to your file. Open in PS, convert to AdobeRGB. Now it's gamma 2.2. Another way to change it is to take your files and run them through a batch process in ImageMagick to convert the gamma to 2.2. Lastly, you can aways take it into PS and use a levels adjustment and move the midpoint to 0.454545. Note that last one has some issues since it seems PS might do some funny stuff with large movements of the gray slider in the levels adjustment. Other graphics programs are probably fine with this though.

You are finding out why I don't think Lightroom is really ideal for adjusting scans that haven't already been adjusted a bit. It's great if you have a more or less color corrected tiff, but if it has issues, like needing to remove the orange mask or adjust gamma, it just doesn't provide the tools that PS does.

Lastly, just a note about locking the exposure and film base. If you do it right, and set Vuescan so that the base is neutral and not orange, you HAVE essentially color balanced the negative. That's how it worked in an enlarger and it's no different here. Unless of course you are shooting your negatives in some light other than the light it was designed for, e.g. shooting a daylight balanced neg indoors under tungsten lighting. Then you might need to do some more white balancing, but that should be a snap for LR, since that's what it's white balancing tools are set up for.

 

[kb2qqm]

i am so glad i happened upon this discussion about Vuescan and Lightroom. This has been making me crazy for 6 months.

I have been doing negative scans on my Epson v500 with the newest weekly download of vuescan. Vuescan scans the negative and it looks great, then, lightroom automatically goes to my scanned in B&W negative scan, and the negtive looks totally blown out.. I,ve been scratching my head..wondering if there was some import filter on, or if something is defective.

I had been scanning in Vuescan and the file would be saved into lightroom 3.6 as a DNG file. Big Problems. so i selected Vuescan to save the file as a 100% jpeg and then both the scanned image and the image displayed in Lightroom were the same, so it had to be a mismatch in how the scanned DNG file was handled on the import into LR. So then I tried just scanning thr B&W negative and saving it as only a 100% TIFF. Problem solved. Can now import the same tonal qualities from Vuescan to Lightroom 3.6 without a problem, and apply correction if needed. The nightmare is over. I will try to post the scans.

http://www.flickr.com/photos/kb2qqm/?uploaded=1&magic_cookie=ed9457d701aac8e0f150ba1041b17447

Greg

 

[metrix x]

No problem here. Vuescan dng and Lightroom 4 work very well together that is if you lock gain and base colour correctly. I never found Lightroom to be perfect for colour balance but it gets you over 90% there which is enough in most cases where it isn't PS can get you the rest of the way. My experience is with colour and B&W negatives. For me it meets the need of non destructive editing and minimum changes to work flow between film and digital.

Using the dng file gives me slightly more range and slightly less noise when it comes to output. It also make scanning easier as I don't have to figure out tone curves before I scan. Raw tiff files without the dng wrapper are difficult to work with.

You might find this from adobe interesting as internally all processing seems to be done at gamma = 1

"Color spaces, color profiles, and tonal response curvesIt’s not necessary for you to understand how Lightroom manages color internally, but the following information may be useful in your workflow.
A color space describes a range or gamut of colors. Various devices in your photographic workflow have different color gamuts in which they can record, store, edit, and output photos. A color profile defines a color space so that Lightroom knows how to manage and convert colors in your photo.
The Library module stores all previews in the AdobeRGB color space. These previews are also used when printing in draft mode. Unless you choose differently in the Soft Proofing panel, the Develop module displays photos in the ProPhotoRGB color space.
A color profile is also defined by a gamma value, or more accurately, its tonal response curve. The tonal response curve defines how tonal values in the raw image are mapped. To provide useful information in the histogram and RGB value display, Lightroom assumes a gamma value of approximately 2.2. More accurately, it uses a tonal response curve similar to the tonal response curve of the sRGB color space.
While Lightroom uses a tonal response curve to provide information for the histogram and RGB values, it manipulates the raw data before it is tone mapped. Working in this linear gamma avoids many of the artifacts that can result in working with a tone-mapped image."