Apple researchers have developed an AI model that dramatically improves extremely dark photos by integrating a diffusion-based image model directly into the camera’s image processing pipeline, allowing it to recover detail from raw sensor data that would normally be lost. Here’s how they did it.
The problem with extreme low-light photos
You’ve probably taken a photo in very dark conditions, which resulted in an image that was filled with grainy, digital noise.
This happens when the image sensor doesn’t capture enough light.
To try to make up for this, companies like Apple have been applying image processing algorithms that have been criticized for creating overly smooth, oil-painting-like effects, where fine detail disappears or gets reconstructed into something barely recognizable or readable.
Enter DarkDiff
To tackle this problem, researchers from Apple and Purdue University have developed a model called DarkDiff. Here’s how they present it in a study called DarkDiff: Advancing Low-Light Raw Enhancement by Retasking Diffusion Models for Camera ISP:
High-quality photography in extreme low-light conditions is challenging but impactful for digital cameras. With advanced computing hardware, traditional camera image signal processor (ISP) algorithms are gradually being replaced by efficient deep networks that enhance noisy raw images more intelligently. However, existing regression-based models often minimize pixel errors and result in oversmoothing of low-light photos or deep shadows. Recent work has attempted to address this limitation by training a diffusion model from scratch, yet those models still struggle to recover sharp image details and accurate colors. We introduce a novel framework to enhance low-light raw images by retasking pre-trained generative diffusion models with the camera ISP. Extensive experiments demonstrate that our method outperforms the state-of-the-art in perceptual quality across three challenging low-light raw image benchmarks.
In other words, rather than applying AI in the post-processing stage, they retasked Stable Diffusion (an open-source model trained on millions of images) to understand what details should exist in dark areas of photos considering their overall context, and integrated it into the image signal processing pipeline.
In fact, their approach introduces a mechanism that computes attention over localized image patches, which helps preserve local structures and mitigates hallucinations like in the image below, where the reconstruction AI changes image content entirely.
... continue reading