Wednesday, November 14, 2012

Demosaicing the Fuji X-Pro1 Part 4


Well, I wasn't expecting to come back to the topic of Fuji, the X-Pro1 and its X-Trans sensor. However, I have been putting a lot of work into the suppression of artifacts when demosaicing. A lot more work than I had intended to, but that's another story. This is for a new product that I hope to release in a few weeks time (several months later than I'd hoped). But I did stumble into a better understanding of the nature of the chroma smearing (or watercolor effect, as it has also become known). The previous posts about Demosaicing the Fuji X-Pro1 are herehere and here.

In previous posts, I compared renderings from Adobe Camera Raw, SILKYPIX and Fuji's in-camera JPEG processing, as well as DCRAW and RPP. Finally, I compared those renderings to renderings from PhotoRaw, both in its "retail" configuration, and in modified form with post demosaic filtering. Practically, DCRAW and RPP were pretty much outclassed -- they use VNG algorithms that generate substantial zipper effects.

In post three, I hypothesized that the chroma smearing effect that you see very visibly in the ACR conversion, and to a lesser extent in the SILKYPIX conversion, was due to filtering, possibly mean filtering post demosaic. I now think that I was probably wrong, or at least partially wrong - the effect is due to filtering, but not mean filtering post demosaic. Rather, it's as a result of filtering during the demosaic process itself.

PhotoRaw, ACR and SILKYPIX

Just for reference, here are the various contenders from previous posts:

PhotoRaw 3.5.4, 400% crop

PhotoRaw does pretty well, but has those pesky artifacts on the paperclips.....

Adobe Camera Raw V7.1 beta

ACR beta 7.1  - Lots of chroma smearing, and the letters are quite desaturated.


SILKYPIX conversion

SILKYPIX - some chroma smearing, saturation down, resolution appears slightly lower than PhotoRaw or ACR


PhotoRaw 3.5.4 plus mean filter, 5 pass, 400% crop

The "PhotoRaw plus mean filter" showed a similar pattern of chroma smearing to SILKYPIX and ACR, but some parts didn't quite match. For both SILKYPIX and ACR, the chroma smearing is more pronounced in specific areas, e.g., in between the upper parts of the "Y", and inside the "A".


Multipass Demosaic Algorithms

As mentioned in the introduction, I've been working on artifact suppression for another product. Now the easiest way to suppress artifacts is not to generate them in the first place, so one of the things I looked at are demosaicing methods that have inherently low levels of chroma artifacts. There are a number of demosaicing algorithms that I'll call "multipass", for lack of a better term. The best known of these is LMMSE (see reference 1 below). LMMSE works, in simplified form, by first performing an initial demosaic in both vertical and horizontal directions, then filtering in the chroma domain, then using the filtered chromas to perform what is in effect a second higher quality demosaic. In this case, higher quality in the sense of fewer visible chroma errors, as the second demosaic is partially based on filtered chroma.

The results of using such a algorithm in a beta version of the new product are shown below. The first crop shows the algorithm using a filter width as it would be for a conventional two by two Bayer array. The second uses exactly the same algorithm, but with double the filter width. Note that unlike the crops above, these two are unsharpened, due to limitations of the beta.



"New Product" beta, multipass demosaic, single filter width, 400% crop, unsharpened



"New Product" beta, multipass demosaic, double filter width, 400% crop, unsharpened

Several thing stand out here (note - you may want to double-click on the crops to get a better view):
  • The multipass algorithm shows fewer artifacts than the "AHD class" algorithm, even with a "normal Bayer" filter width.
  • However, with a wider filter two things happen. Firstly, artifacts are reduced, as we'd expect - the X-Trans sensor has sparse colored photo sites, so its logical that we'd have to filter more heavily to get this particular algorithm to perform. Secondly, the crop starts to show the characteristic chroma smear pattern that the SILKYPIX conversion shows, as well as the slight loss of resolution and saturation that the SILKYPIX conversion shows.  In fact, the conversion starts to look uncannily like SILKYPIX, a lot more so that was the case for the mean filter I originally suspected was involved.
  • Extrapolating, you can easily see how, with an ever wider filter, ACR would generate the rendering it does.


Conclusion

It's highly likely that both SILKYPIX and ACR use some form of multipass algorithm, and that the chroma smearing is as a result of the need to adapt the algorithm to the sparse colored photo sites in the X-Trans sensor. ACR appears to be using a very wide filter, probably indication the whatever methods are being used in the initial and secondary demosaic are not well suited to the X-Trans sensor. Note that it's highly unlikely that either ACR or SILKYPIX are using LMMSE in anywhere near it's text book form (neither is the "New Product" beta used above, btw). The text book form depends on using color data to improve the initial green demosaic; given the X-Trans sensors relatively large number of green photo sites and low number of red or blue photo sites, that would not work well.



References:

1. L. Zhang and X. Wu, Color demosaicking via directional linear minimum mean square estimation, IEEE Trans. on Image Processing, vol. 14, pp. 2167-2178, Dec. 2005.

8 comments:

Vladimir said...

It seems like a problem without a real solution.
Not sure if I am understanding this correctly, but would a higher MP count X-trans sensor scaled down (binning) be better?

Sandy said...

I don't know that there isn't a solution. There are numbers of advanced interpolation methods, e.g., projection schemes, etc, etc. One or more of them would probably work better than what we have now. The problem is that nobody actually knows which will work well on the X-Trans sensor (outside Fuji anyway, and they aren't telling), and testing all of them is prohibitively costly.

Binning would probably improve the artifact situation, but the question then is what's the advantage of the sensor....

Ario Arioldi said...

Sandy,
is there any evidence that Fuij has identified an interpolation algorithm substantially better then others?
Judging either from Silkpix or from the OOC jpeg I personally don't see it.

Sandy said...

Ario,

It's difficult to tell with any certainty. I'd agree there's no sign that either SILKYPIX or the in-camera processing are hugely ahead of anything else. That said, SILKYPIX and the in-camera JPEG processor are still somewhat better than anything else out there, at least as regards freedom from artifacts, although both lose some resolution doing so.

My feeling is that SILKYPIX is using a fairly conventional algorithm, just well tuned to the X-Trans sensor. It's not clear what's going on with the in camera JPEG processing.

Yves Ste-Marie said...

There is a lot of word which have been written on what is the best way to treat a RAF file, Lightroom, Sylkypix, etc. None is as good for filtering post demosaic then the in camera processing done for the in camera JPG.

So why not simply make a request to Fujifilm for a new type of RAF file, a RDF file (Raw Demosaic File). That way we would benefit from the RAW file processing and the filtering demosaicing of the in camera algorithm.

And there is no reason to have the tree in camera file types RAW, JPG and RDF.

Sandy said...

Yves,

There is a rumor that Fuji are going to add TIFF output to X-Pro. That would achieve the same result.

nixda said...

Hi Sandy,

First, many thanks to openly describe your valiant efforts so well.

Now, one thing that struck me is when you said (paraphrased) that "testing every possibility would be prohibitively expensive". I am sure all of us from the enthusiast community would be more than willing to test your various approaches, play with settings, etc. and report the results back to you, if you provided us with the tools. You wouldn't have to do all the work yourself, just sift through all the responses (which might be taxing as well...).

The Hugin community is a good example for such a community-driven effort, as there are hundreds of others, mostly in academia.

Regarding speed of RAW converters: I don't think users of your software would mind if it was slow. As long as the result is good. Of course, that requires a version for Mac OSX, or other Unix flavors, not just for iOS. Even if it was only command-line driven. Any chance of ever providing such a version?

Sandy said...

Nixda,

The problem is the writing of the code, not so much the actual testing. There is a lot of code for the various demosiacing algos out there, e.g., from the academic community, either in C or Matlab form, but about 99% of that assumes a 2x2 layout, so has to be extensively modded.

The "new product" I referred to in the post above is a Mac OS X program. There will be a public beta available soon - watch this space....