A #machinelearning library written in pure LaTeX

Web: https://t.co/uwAn2LrV8z Paper: https://t.co/2IBwmF9Jb2 Source: https://t.co/2IBwmF9Jb2 (click "TeX source")

This has been a massively entertaining and challenging side project with @JADGardner and @WillRowan3 over the past year and (subject to rigorous peer review) will be appearing at @sigbovik 2025.

@overleaf becomes your cloud compute provider. This should level the playing field within the community as both industry and academia will have to work within the same compute limits.

NeuRaLaTeX brings many side benefits. Your paper source code is also your method source code. No more "Code coming soon" on github. If the paper is on arxiv, the paper source link *is* the method source code! No need to remember those silly git commands anymore!

In case your arXiv submission is timing out, we've also implemented checkpointing so you can include your trained model weights as a text file with your paper source.

In a NeuRaLaTeX paper, when you compile your PDF, a neural network is constructed, trained and evaluated with all results and figures generated dynamically. We've worked hard on efficiency and the NeuRaLaTeX paper only took 48 hours to compile.

Our neural network library extends the autograd engine to create neurons, linear layers and MLPs. Constructing an MLP and making a forward pass is as easy as this.

Just like @karpathy's micrograd, NeuRaLaTeX implements backpropagation (reverse-mode autodiff) over a dynamically built DAG. However, while micrograd uses 150 lines of Python, NeuRaLaTeX uses around 1,100 line of pure LaTeX, making it about 700% better.

Not only that, but LaTeX has elegant and intuitive programming syntax. For example, creating a new Value object only uses the word "expand" four times: \expanded{ \noexpand\pgfoonew\expandafter\noexpand \csname #2\endcsname= new Value(\newdata,{\selfid,\otherid},*,0) }

Wait, what? Well, LaTeX is itself a Turing-complete programming language. When you "compile" a LaTeX document, really you are executing a program written in LaTeX. This program need not only format your paper contents into a PDF but can also perform useful computation.

So, we wrote a neural net library entirely in LaTeX...

Good morning all!

We end up with convincingly de-lit diffuse albedo, detailed geometry and realistic HDR lighting estimates. And just look at those differentiable shadows! This was a very fun project to work on and an epic effort from @JADGardner! 12/12

In practice, we soften this binary depth comparison with a smooth sigmoid, whose width is a learnable parameter. This allows soft blurry shadows early in training becoming sharp as the training converges. 11/12

We can use the DDF for shadow testing. For a given scene point and light direction, we compute the inward pointing depth with the DDF and compare this to the scene point to bounding sphere distance. If this distance is larger, the point is shadowed. 10/12

We encourage the DDF to be geometry-consistent with the underlying scene SDF. A consequence is that the appearance of a shadow can, via the DDF, encourage the SDF to introduce the occluder into the scene geometry. Even unobserved shadow casters can be reconstructed! 9/12

Our underlying geometric representation is an SDF. Computing depth from lighting directions is still expensive, requiring sphere tracing. So, we train a Directional Distance Function on a scene-bounding sphere. Looking in on the scene, this lets us query distance to scene. 8/12

We were inspired by classical shadow mapping. Here, a depth map is rendered from the light source. Then the points visible from the viewer are depth tested against the shadow map. If there is an occluder, the depths will differ significantly. 7/12

Shadows are a key feature of the appearance of outdoor scenes since the brightest parts of the sky are often occluded by buildings, causing soft shadows. Exhaustively volume rendering from all scene points in all lighting directions is too expensive. 6/12