Ben Mildenhall

I am a research scientist at Google Research, where I work on problems in computer vision and graphics. I received my PhD from UC Berkeley in 2020, where I was advised by Ren Ng and supported by a Hertz fellowship.

In the summer of 2017, I was an intern in Marc Levoy's group in Google Research. In the summer of 2018, I worked with Rodrigo Ortiz-Cayon and Abhishek Kar at Fyusion. I did my undergrad at Stanford University and worked at Pixar Research in the summer of 2014.

Email  /  CV  /  Bio  /  Google Scholar  /  Twitter  /  GitHub

DreamFusion: Text-to-3D using 2D Diffusion
Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall
ICLR, 2023 (Outstanding Paper Award)
project page / arXiv / gallery

We optimize a NeRF from scratch using a pretrained text-to-image diffusion model to do text-to-3D generative modeling.

Fast and High-quality Image Denoising via Malleable Convolutions
Yifan Jiang, Bartlomiej Wronski, Ben Mildenhall, Jonathan T. Barron, Zhangyang Wang, Tianfan Xue
ECCV, 2022
project page / arXiv

We denoise images efficiently by predicting spatially-varying kernels at low resolution and using a fast fused op to jointly upsample and apply these kernels at full resolution.

NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images
Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan Barron
CVPR, 2022 (oral)
project page / arXiv / video / code

We train RawNeRF directly on linear raw camera images, enabling new HDR view synthesis applications and greatly increasing robustness to camera noise.

Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, Peter Hedman
CVPR, 2022 (oral)
project page / arXiv / video / code

We extend mip-NeRF to produce photorealistic results on unbounded scenes.

Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, Pratul Srinivasan
CVPR, 2022 (Best Student Paper Honorable Mention)
project page / arXiv / video / code

Explicitly modeling reflections in NeRF produces realistic shiny surfaces and accurate surface normals, and lets you edit materials.

Block-NeRF: Scalable Large Scene Neural View Synthesis
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T. Barron, Henrik Kretzschmar
CVPR, 2022 (oral)
project page / arXiv / video

We build city-scale scenes from many NeRFs, trained using millions of images.

RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs
Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, Noha Radwan
CVPR, 2022 (oral)
project page / arXiv / video

We regularize unseen views during optimization to enable view synthesis from as few as 3 input images.

Zero-Shot Text-Guided Object Generation with Dream Fields
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, Ben Poole
CVPR, 2022
project page / arXiv / video / code

Supervising the CLIP embeddings of NeRF renderings allows us to generate 3D objects from text prompts alone.

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views
Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul Srinivasan, Matthias Nie├čner
CVPR, 2022
arXiv / video

We apply dense depth completion techniques to freely-available sparse stereo data to guide NeRF reconstructions from few input images.

Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan
ICCV, 2021 (Best Paper Honorable Mention)
project page / arXiv / video / code

We prefilter the positional encoding function and train NeRF to generate anti-aliased renderings.

Baking Neural Radiance Fields for Real-Time View Synthesis
Peter Hedman, Pratul Srinivasan, Ben Mildenhall, Jonathan T. Barron, Paul Debevec
ICCV, 2021 (oral)
project page / arXiv / video / demo

We bake a trained NeRF into a sparse voxel grid of colors and features in order to render it in real-time.

Learned Initializations for Optimizing Coordinate-Based Neural Representations
Matthew Tancik*, Ben Mildenhall*, Terrance Wang, Divi Schmidt, Pratul Srinivasan, Jonathan T. Barron, Ren Ng
CVPR, 2021 (oral)
project page / arXiv / video / code

We use meta-learning to find weight initializations for coordinate-based MLPs that allow them to converge faster and generalize better.

NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis
Pratul Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, Jonathan T. Barron
CVPR, 2021
project page / arXiv / video

We recover relightable NeRF-like models using neural approximations of expensive visibility integrals, so we can simulate complex volumetric light transport during training.

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
Matthew Tancik*, Pratul Srinivasan*, Ben Mildenhall*, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, Ren Ng
NeurIPS, 2020 (spotlight)
project page / arXiv / code

We demonstrate that composing fully-connected networks with a simple Fourier feature mapping allows them to learn much high frequency functions.

Neural Reflectance Fields for Appearance Acquisition
Sai Bi*, Zexiang Xu*, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milos Hasan, Yannick Hold-Geoffroy, David Kriegman, Ravi Ramamoorthi
arXiv, 2020

We recover relightable NeRF-like models by predicting per-location BRDFs and surface normals, and marching light rays through the NeRV volume to compute visibility.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Ben Mildenhall*, Pratul Srinivasan*, Matthew Tancik*, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
ECCV, 2020 (Best Paper Honorable Mention)
project page / arXiv / video / talk / code / two minute papers / papers with code / wired

We optimize a simple fully-connected network to represent a single scene as a volume, then use volume rendering to do view synthesis.

Deep Multi Depth Panoramas for View Synthesis
Kai-En Lin, Zexiang Xu, Ben Mildenhall, Pratul Srinivasan, Yannick Hold-Geoffroy, Stephen DiVerdi, Qi Sun, Kalyan Sunkavalli, Ravi Ramamoorthi
ECCV, 2020
arXiv / video

We represent scenes as multi-layer panoramas with depth for VR view synthesis.

Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination
Pratul Srinivasan*, Ben Mildenhall*, Matthew Tancik, Jonathan T. Barron, Richard Tucker, Noah Snavely
CVPR, 2020
project page / arXiv / video / code

We predict a volume from an input stereo pair that can be used to calculate incident lighting at any 3D point within a scene.

StegaStamp: Invisible Hyperlinks in Physical Photographs
Matthew Tancik*, Ben Mildenhall*, Ren Ng
CVPR, 2020
project page / arXiv / video / code

We can hide hyperlinks in natural images to create aesthetically pleasing barcodes.

Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines
Ben Mildenhall*, Pratul Srinivasan*, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, Abhishek Kar
project page / arXiv / video / code

We develop and analyze a deep learning method for rendering novel views of complex real world scenes.

Unprocessing Images for Learned Raw Denoising
Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, Jonathan T. Barron
CVPR, 2019 (oral)
project page / arXiv

We can learn a better denoising model by processing and unprocessing images the same way a camera does.

Burst Denoising with Kernel Prediction Networks
Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, Robert Carroll
CVPR, 2018 (spotlight)
project page / arXiv / code

We train a network to predict linear kernels that denoise bursts of raw linear images.

DiffuserCam: Lensless Single-exposure 3D Imaging
Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, Laura Waller
Optica, 2018
project page / arXiv

Using a diffuser instead of a lens lets you recover 3D in a single exposure.


Approximations for the distribution of microflake normals
Nelson Max, Tom Duff, Ben Mildenhall, Yajie Yan
The Visual Computer, 2017

We precompute microflake approximations to make rendering large meshes at a distance more efficient.


Controlling Procedural Modeling Programs with Stochastically-Ordered Sequential Monte Carlo
Daniel Ritchie, Ben Mildenhall, Noah D. Goodman, Pat Hanrahan

We improve control over the output of highly-variable procedural modeling programs by using SOSMC to provide incremental feedback on partially-generated models.

Course Projects

Extending the PBRT Renderer to Support Volumetric Light Sources
Grand prize in CS384B rendering competition, Spring 2013

Adding support for multicolored, nonhomogeneous, emissive volumes in PBRT 2's path tracing integrator.


Ray Tracer
CS148 assignment, Fall 2012

Implemented a ray tracer with reflections, refractions, soft shadows, and depth of field.


CS184 - Summer 2020 (Co-instructor)

CS184 - Spring 2017 (GSI)

CS184 - Spring 2016 (GSI)


Rachmaninoff - Etude-Tableau Op. 39 No. 5
Encore after a salon recital, Spring 2019

Rachmaninoff - Rhapsody on a Theme of Paganini
With Stanford Symphony Orchestra, Spring 2015

Brahms - Piano Quintet
Recital at Stanford, Winter 2015

Yep it's another Jon Barron website.
Last updated March 2023.