Raw files and Original Data
Original in-camera sensor assembly data is unavailable from consumer digital cameras.
In consumer cameras digital imaging the data initially written by the ADC in-camera is rarely, if ever, written to the the raw file we use in post-production image rendering. This violates a common-sense principle of data analysis – never modify original data an any way. [1]
All camera makers have a problem. Their raw data will be used by a different third-party image-rendering software venders. Camera makers have no control over how completely a software vendor will model their data.[2] This means camera makers have to modify the original data to minimize variations in image perception that could damage their brand.
There is not much detailed information about changes camera makers apply to the original data.
Some known modifications to the original data include:
. various types of noise filtering – especially at very high camera ISO settings.
. dark frame subtraction to minimize artifacts present in long exposures
. pixel binning in raw file data from CMOS sensors.[3]
. Nikon mathematically replaces all negative digital numbers (DN) in their original data with zeros.
. Canon’s C-RAW option (.CR2 and .CR3 files) uses proprietary lossy compression to reduce file size. Original data is discarded and unrecoverable. This could be the ultimate abuse of the word raw in photography since a reasonable person would assume C-RAW is similar to lossless compression.
. when a Leica DNG is created using coded lens on a M8 or M9 body. the original DN values are irreversibly modified to correct for lens vignetting and hue infidelity.
What about lens correction parameters?
Let’s address three types of lens corrections: corrections for geometric distortions, for reduced brightness (luminance) and hue infidelity at the edges of the frame.
Geometric distortion is corrected using empirical parameters provided by the lens manufacturer or by independent sources. Leica lenses designed for film cameras have very low geometric distortion. Many Leica M, LTM and S series lenses do not require distortion correction. Leica coded lenses never applied geometric distortion corrections to DNG files from the M8 or M9. DNG files from Leica lenses designed specifically for digital cameras (e.g Q series) do contain geometric distortion correction parameters.
Since vignetting affects signal amplitude, straightforward DN corrections based on empirical measurements can be made to the original data amplitudes. It also possible to make these corrections during image demosaicking which means the ordinal data is not changed. Instead the demosaicking model includes amplitude correction parameters. In both cases amplitude correction parameters can be identically applied to R, G and B DN values according to their distance from the center of the frame. Hue rendering inaccuracies are addressed by using different amplitude correction parameters for R, G and B DN values.
Of course it is possible for photographers to create their own parameter correction estimates for any lens. Third-party software exists to simplify this task.
Leica decided to use the open-source DNG raw file format. Before format version 1.3 (2009), DNG files did not support lens x correction parameters. For this reason Leica’s lens coding applies vignetting and hue amplitude corrections to the DNG file. The original DNs are irreversibly modified. DNG format versions 1.3 and later do support storage of lens correction parameters. I so not know if or when Leica abandoned modifying original M camera DNs and switched to storing lens coding correction parameters as DNG file operation codes for the M240 or newer M bodies.
1/ Photography is identical to all empirical measurements in that collected data is used to compte parameter estimates. The parameters of interest are those in the mathematical model(s) used to render an image from the data. In our case these are a spatial array of the absolute energy accumulated by a photo-diode in the individual pixels. When a mathematical model accounts for all the information in the data, there is no need to modify the original data. This is rare for a variety of practical reasons.
2/ For example, a demosaicking model that computes a rendered image may assume there was no DC offset (bias) in the data. If a DC offset is present, a demosaicking model without a DC offset parameter will produce an image that is inferior to a model that includes one. This can be a trivial (DC offset is small) or serious (DC offset is high) issue.
3/ At this time the Leica M11 is the only consumer CMOS camera with this feature. This is different than the most modifications of original data because the signal and noise in individual DNs are averaged together. In other fields digital signal averaging occurs in memory during data acquisition. So, the original signal data is always an average. However pixel binning irreversibly decreases a sensor’s maximum possible resolution information content.