the gradient.030 · MITSUBA 3 · DIFFERENTIABLE RENDERING · 2022

Pictura quaerit causam suam.

— THE PICTURE SEEKS ITS OWN CAUSE —

Every renderer in this stack runs forward: scene goes in, image comes out. Mitsuba 3 is the renderer that runs backward. Given an image, it recovers the scene that produced it — by computing the gradient of the rendered pixels with respect to the scene parameters and stepping toward the answer. The renderer is no longer a function; it is a derivative.

In [1]:

import mitsuba as mi
import drjit   as dr
mi.set_variant("cuda_ad_rgb")   # JIT-compiled · differentiable · GPU

scene  = mi.load_file("./scene.xml")
target = mi.Bitmap("./target.exr")

θ = mi.traverse(scene)                    # the parameter graph
for key in ["sphere.bsdf.albedo", "light.intensity", "sphere.position"]:
    dr.enable_grad(θ[key])               # mark as differentiable

In [2]:

opt = mi.ad.Adam(lr=0.025)
opt.load(θ)

for i in range(512):
    img  = mi.render(scene, params=θ, spp=8)     # forward render
    loss = dr.mean(dr.sqr(img - target))         # L2 image loss
    dr.backward(loss)                              # reverse-mode AD
    opt.step()                                     # θ ← θ - η · ∇θ ℒ
    opt.update(θ)
    log(i, float(loss))

Out[2]:

TARGET — I_refthe photograph

FROZEN · 0.000e+00 grad SPP 1024

RENDER — I_θthe recovery

iter 000 · ∇ active SPP 8

ℒ = ‖I_θ − I_ref‖² · image L₂ loss 1.0000e+00

θ — scene parameters under optimization

parameter	target	current	∇ θ ℒ	Δ

θ_t+1 = θ_t − η · ∇_θ ℒ(R(θ_t), I_ref) — Equation 1 · the rendering gradient step —

Where R is the path-tracing operator, ℒ is the loss between rendered and reference image, and η is the step size. The renderer R is differentiable by construction. No finite differences. No black-box optimization. The gradient flows through every refraction, every BSDF, every Monte Carlo sample, all the way back to the parameter that created it.

Cell · Production Configuration masthead — Mitsuba 3.x

AuthorWenzel Jakob · EPFL · Realistic Graphics Lab

First Ship2010 · Mitsuba 0.1 · 2022 · 3.0

EngineDr.Jit · JIT · vectorized · auto-diff

Variantsscalar · llvm · cuda · spectral · polarized · ad

LicenseBSD-3-Clause · academic

Native ToResearch · SIGGRAPH ground truth · inverse rendering

— Bibliography (Selected) —

Jakob, W., Speierer, S., Roussel, N., et al. Mitsuba 3: a retargetable forward and inverse renderer. SIGGRAPH. 2022.
Nimier-David, M., Vicini, D., Zeltner, T., Jakob, W. Mitsuba 2: a retargetable forward and inverse renderer. ACM Trans. Graph. 2019.
Jakob, W. Mitsuba 0.6 — a research-oriented physically-based renderer. 2010 — 2014.
Pharr, Jakob, Humphreys. PBR-Book v3: Physically-Based Rendering, MIT Press. 2018. Mitsuba is the working code for the textbook.
Vicini, D., Speierer, S., Jakob, W. Path replay backpropagation: differentiating light paths using constant memory. SIGGRAPH. 2021.
Loubet, G., Holzschuch, N., Jakob, W. Reparameterizing discontinuous integrands for differentiable rendering. SIGGRAPH Asia. 2019.
Zeltner, Speierer, Georgiev, Jakob. Monte Carlo estimators for differential light transport. SIGGRAPH. 2021.
Vicini, Jakob, Speierer. Differentiable signed distance function rendering. SIGGRAPH. 2022.

— Version History · Mitsuba —

2010	0.1	Wenzel Jakob's PhD work at Cornell. Forward only.
2014	0.6	Definitive forward-rendering reference. Used for SIGGRAPH ground truth for a decade.
2019	2.0	Retargetable variants. Compile once for spectral, polarized, RGB, scalar, GPU.
2022	3.0	Built on Dr.Jit. Differentiable rendering becomes a first-class operation.
2023	3.4	Path replay backpropagation. Constant-memory reverse-mode AD.
2024	3.5	Reparameterized discontinuous integrands. Edge gradients become tractable.

render. measure. step. repeat.

The renderer figures out the scene by failing slightly less, every frame.

— θ ← θ − η · ∇_θ ℒ —

— § I · Authoring · Renderers / Research —

mitsuba_3.ipynb · MMXXVI

projectlavos-bauhaus · 030