NeRF in the Dark:
High Dynamic Range View Synthesis from Noisy Raw Images
CVPR 2022 (Oral Presentation)

Abstract

Neural Radiance Fields (NeRF) is a technique for high quality novel view synthesis from a collection of posed input images. Like most view synthesis methods, NeRF uses tonemapped low dynamic range (LDR) as input; these images have been processed by a lossy camera pipeline that smooths detail, clips highlights, and distorts the simple noise distribution of raw sensor data. We modify NeRF to instead train directly on linear raw images, preserving the scene's full dynamic range. By rendering raw output images from the resulting NeRF, we can perform novel high dynamic range (HDR) view synthesis tasks. In addition to changing the camera viewpoint, we can manipulate focus, exposure, and tonemapping after the fact. Although a single raw image appears significantly more noisy than a postprocessed one, we show that NeRF is highly robust to the zero-mean distribution of raw noise. When optimized over many noisy raw inputs (25-200), NeRF produces a scene representation so accurate that its rendered novel views outperform dedicated single and multi-image deep raw denoisers run on the same wide baseline input images. As a result, our method, which we call RawNeRF, can reconstruct scenes from extremely noisy images captured in near-darkness.

overview

Overview Video

HDR View Synthesis

RawNeRF output images in linear HDR color space, so its renderings can be retouched like any raw photograph. Here we change the exposure by scaling the linear image values before applying HDRNet (Gharbi et al. 2017) to produce a tonemapped low dynamic range output. We can also use the linear colors to render synthetic defocus effects with correctly saturated "bokeh" highlights.

Denoising

Training directly on raw data effectively turns RawNeRF into a multi-image denoiser capable of combining information from tens or hundreds of input images. This robustness to noise means that we can use RawNeRF to reconstruct scenes captured in the dark.

Citation

Acknowledgements

We thank our colleagues on the HDR+ team for answering our questions about how cameras and tonemapping work. Bart Wronski provided helpful feedback on image processing pipeline terminology.
The website template was borrowed from Michaël Gharbi.