Differentiable Programming from Scratch

Differentiable Programming from Scratch Differentiable programming has become a hot research topic, and not only due to the popularity of machine learning frameworks like TensorFlow, PyTorch, and JAX. Many fields apart from machine learning are finding differentiable programming to be a useful tool for solving optimization problems. In computer graphics, differentiable rendering, differentiable physics, and neural representations are all gaining popularity. This article received an honorable mention in 3Blue1Brown’s Summer of Math Exposition 2! Prerequisites Differentiation It all starts with the definition you learn in calculus class: \[ f^{\prime}(x) = \lim_{h\rightarrow 0} \frac{f(x + h) - f(x)}{h} \] In other words, the derivative computes how much $$f(x)$$ changes when $$x$$ is perturbed by an infinitesimal amount. If $$f$$ is a one-dimensional function from $$\mathbb{R} \mapsto \mathbb{R}$$, the derivative $$f^{\prime}(x)$$ returns the slope of the graph of $$f$$ at $$x$$. ... Read full article.