The model, called SHARP, can reconstruct a photorealistic 3D scene from a single image in under a second. Here are some examples.
SHARP is just awesome
Apple has published a study titled Sharp Monocular View Synthesis in Less Than a Second, in which it details how it trained a model to reconstruct a 3D scene from a single 2D image, while keeping distances and scale consistent in real-world terms.
Here’s how Apple’s researchers present the study:
We present SHARP, an approach to photorealistic view synthesis from a single image. Given a single photograph, SHARP regresses the parameters of a 3D Gaussian representation of the depicted scene. This is done in less than a second on a standard GPU via a single feedforward pass through a neural network. The 3D Gaussian representation produced by SHARP can then be rendered in real time, yielding high-resolution photorealistic images for nearby views. The representation is metric, with absolute scale, supporting metric camera movements. Experimental results demonstrate that SHARP delivers robust zero-shot generalization across datasets. It sets a new state of the art on multiple datasets, reducing LPIPS by 25–34% and DISTS by 21–43% versus the best prior model, while lowering the synthesis time by three orders of magnitude.
In a nutshell, the model predicts a 3D representation of the scene, which can be rendered from nearby viewpoints.
A 3D Gaussian is basically a small, fuzzy blob of color and light, positioned in space. When millions of these blobs are combined, they can recreate a 3D scene that looks accurate from that specific vantage point.
To create this kind of 3D representation, most Gaussian splatting approaches require dozens or even hundreds of images of the same scene, captured from different viewpoints. Apple’s SHARP model, by contrast, is able to predict a full 3D Gaussian scene representation from a single photo in one forward pass of a neural network.
To achieve this, Apple trained SHARP on large amounts of synthetic and real-world data, enabling it to learn common patterns of depth and geometry across multiple scenes.
As a result, when given a new photo, the model estimates depth, refines it using what it has learned, and then predicts the position and appearance of millions of 3D Gaussians in a single pass.
... continue reading