What Is a Diffusion Model, and How Does It Generate Images

Most of today’s most striking AI-generated images come from a technique with a genuinely counterintuitive origin story: models that learn to create images by first learning how to destroy them. Diffusion models are now the dominant approach behind leading image generators, and the underlying idea — train a model to reverse noise, step by step, until a clear image emerges — is one of the more elegant ideas in recent machine learning.

The Forward Process: Adding Noise on Purpose

Training a diffusion model starts by taking real images and progressively corrupting them, adding random visual noise in many small steps until the original image is completely obscured, leaving what looks like pure static. This “forward” noising process is simple and doesn’t require any learning at all — it’s a fixed, well-understood mathematical procedure applied identically to every training image.

The Reverse Process: Learning to Undo It

The actual learning happens on the reverse direction: the model is trained to predict, at each step, what noise was just added, so that subtracting its prediction moves the corrupted image slightly back toward a cleaner version. Trained across millions of images and every step of the noising process, the model becomes skilled at this specific, narrow task: given a somewhat noisy image, estimate what noise is present and remove a bit of it.

From Denoising to Generation

Generation flips this learned denoising process into a creative tool. Instead of starting from a real, gradually-noised image, the process starts from pure random noise — essentially static with no image hidden inside it — and repeatedly applies the trained model’s denoising step, gradually sculpting that random noise into a coherent image over many iterations. Because the model learned to remove noise in a way that’s consistent with how real images are structured, this iterative denoising process from pure randomness tends to converge on plausible, coherent-looking images rather than more noise.

Where Text Prompts Actually Enter the Process

Text-to-image diffusion models add a text-understanding component that guides each denoising step according to a written prompt, nudging the emerging image toward content matching the described scene at every single step of the process rather than applying the instruction only once. This is why the composition, subject matter, and style of a generated image can shift meaningfully based on precise prompt wording — the guidance signal shapes the image gradually, across dozens of iterative refinement steps, not in one shot.

Why This Approach Beat Earlier Generative Techniques

Diffusion models became the dominant image-generation approach partly because the iterative, step-by-step refinement process tends to produce more stable, higher-quality, more diverse results than earlier generative approaches, which often struggled with training instability or produced a narrower range of outputs. The gradual, many-step nature of diffusion also gives the process more opportunities to correct course, refining rough overall composition in early steps and fine detail in later ones, in a way that’s proven more reliable to train at scale.

The Real Cost: Many Steps Means Real Compute

The iterative nature that makes diffusion models effective also makes them computationally demanding: generating a single image can require dozens or even hundreds of individual denoising steps, each requiring a full pass through the underlying neural network. This is a major reason ongoing research has focused heavily on reducing the number of steps needed, through techniques that let the model make larger, more efficient jumps toward a clean image, since faster generation with comparable quality directly translates into cheaper, more responsive image-generation products.

Beyond Static Images

The same diffusion principle has extended well beyond still images into video generation, audio synthesis, and even some approaches to generating text and structured data, wherever a system benefits from iterative, gradual refinement rather than committing to an output in one pass. This has made diffusion less a single-purpose image technique and more a genuinely general-purpose generative modeling approach, adapted to whatever type of data a particular application needs to create.

Latent Diffusion: Working in a Compressed Space

A key efficiency improvement adopted by most modern image diffusion systems is running the entire noising-and-denoising process in a smaller, compressed “latent” representation of the image rather than on full-resolution pixels directly. A separate, previously trained component compresses images into this smaller latent space and decompresses them back into full-resolution pixels only at the very end, which meaningfully reduces the computation needed for every single one of the many denoising steps in between, without much sacrifice in final image quality.

Further reading: See our guides to AI Image Generators: How Text-to-Image Models Actually Work and How Multimodal AI Models See, Hear, and Understand the World. For background, see Wikipedia’s entry on Diffusion model.

Want to contribute to Daily AI News? We welcome guest posts, research write-ups, and press releases from the AI community. Write for us →