Edit: we are very pleased to have been awarded the Best Paper Presentation & Best Paper Runner-Up awards after my presentation of this work at I3D 2024! The presentation recording will be available shortly.
In our latest work at I3D 2024, we show how to transform any rasterizer into a differentiable one with minimal engineering and zero external dependencies. Optimize game assets and volumes, train 3D Gaussian Splats, etc. directly within your existing engine. Compute shaders is all you need!
Differentiable rendering is great but reimplementing engine-specific algorithms from scratch in PyTorch or new fancy shading languages is painful. Here, we jointly optimize a Catmull-Clark subdivision surface with displacement mapping and PBR textures in a classic renderer:
By implementing our method in Unity, we can thus achieve fully cross-platform differentiable rendering, running even on low-end mobile phones:
Abstract
We show how to transform a non-differentiable rasterizer into a differentiable one with minimal engineering efforts and no external dependencies (no Pytorch/Tensorflow). We rely on Stochastic Gradient Estimation, a technique that consists of rasterizing after randomly perturbing the scene’s parameters such that their gradient can be stochastically estimated and descended. This method is simple and robust but does not scale in dimensionality (number of scene parameters). Our insight is that the number of parameters contributing to a given rasterized pixel is bounded. Estimating and averaging gradients on a per-pixel basis hence bounds the dimensionality of the underlying optimization problem and makes the method scalable. Furthermore, it is simple to track per-pixel contributing parameters by rasterizing ID- and UV-buffers, which are trivial additions to a rasterization engine if not already available. With these minor modifications, we obtain an in-engine optimizer for 3D assets with millions of geometry and texture parameters.