AI & Technology

How AI Image Generation Works: Stable Diffusion, Flux, and Midjourney Explained

· 15 min read

AI image generators produce stunning visuals from a few words of text. But how do they actually work? This guide explains the core technology behind modern diffusion models in plain language — covering what happens between your text prompt and the final image, and how tools like Stable Diffusion, Flux, Midjourney, and DALL-E differ.

A Brief History of AI Image Generation

The modern era of AI image generation emerged from decades of machine learning research, but the breakthrough that changed everything came from a technique called generative adversarial networks (GANs), introduced by Ian Goodfellow and colleagues in 2014. GANs pit two neural networks against each other — a generator that creates images and a discriminator that tries to tell generated images from real ones. The two networks improve together, eventually producing remarkably realistic images.

GAN-based systems like StyleGAN could generate photorealistic human faces, but they struggled with compositionality — generating specific scenes, objects, and styles from arbitrary text descriptions. The pivot point came in 2021 and 2022, when diffusion models became the dominant paradigm. Research from OpenAI (DALL-E), Google (Imagen), and Stability AI (Stable Diffusion) demonstrated that diffusion models could generate high-quality, text-guided images at an unprecedented scale.

The open release of Stable Diffusion in August 2022 was particularly transformative. For the first time, anyone with a consumer GPU could run a state-of-the-art image generation model locally, without paying per image, without cloud dependency, and with full control over the generation process.

Since then, the field has evolved rapidly: Stable Diffusion XL (SDXL, 2023), Stable Diffusion 3 (2024), and Black Forest Labs' Flux models (2024) have each brought significant quality improvements. Meanwhile, Midjourney, Adobe Firefly, and similar cloud-based services have made AI image generation accessible to non-technical users through polished interfaces.

What is a Diffusion Model?

A diffusion model learns to generate images by learning to reverse a process of gradually adding noise to an image. Here is the intuition:

Forward process (training time): Take a real photograph. Gradually add random Gaussian noise to it in many small steps — step 1, step 2, step 3 — until after perhaps 1000 steps, the original image is completely unrecognizable. It is now just random noise.

Reverse process (what the model learns): The neural network is trained to predict: given a noisy image at step t, what does the slightly less noisy image at step t-1 look like? If the model can accurately denoise one step at a time, it can start from pure random noise (step 1000) and work backwards to a clean image (step 0).

The model learns this denoising skill across millions of image-text pairs during training. It sees noisy versions of real images at every noise level, and learns to denoise them. The key insight is that the model implicitly learns the statistical distribution of natural images — which pixel patterns are common, which are unlikely, and what visual structures tend to co-occur.

At generation time, you give the model a purely random noise image. It applies the denoising step 20–100 times (depending on your settings), each time making the image slightly cleaner and more structured, until a coherent image emerges. The text prompt guides this denoising process — the model is steered toward images whose appearance is consistent with the text description.

Latent Diffusion Models — Why They're Fast

Early diffusion models operated directly in pixel space — every denoising step computed predictions for every pixel in the final image. This was extremely computationally expensive. A 512×512 image has 786,432 pixels (each with three color channels), and running 1000 denoising steps on all of them required enormous GPU resources.

Latent diffusion models (LDMs), introduced in Rombach et al. (2022) and the basis for Stable Diffusion, solve this by working in a compressed latent space instead of pixel space.

An autoencoder first compresses images into a much smaller latent representation — typically 8× smaller in each spatial dimension. A 512×512 image becomes a 64×64 latent representation with 4 channels. The diffusion process runs in this compressed space, which is 64 times smaller than pixel space, making each denoising step much faster.

At the end of generation, the decoder (the other half of the autoencoder) upsamples the latent representation back to full pixel resolution. This decode step happens once and is fast.

The result: latent diffusion models can generate high-resolution images in seconds on a consumer GPU, compared to minutes or longer for pixel-space diffusion models.

How Text Prompts Guide the Image

The bridge between your text prompt and the image is a text encoder. The most common text encoder used in Stable Diffusion 1.x and 2.x models is CLIP (Contrastive Language-Image Pre-training), developed by OpenAI.

CLIP was trained on hundreds of millions of image-text pairs scraped from the internet. It learned to map both images and text into a shared embedding space — a high-dimensional mathematical space where semantically similar concepts end up close together, regardless of whether they are expressed as images or words. "A red apple" and an actual photograph of a red apple would have nearby positions in this space.

During image generation, your text prompt is passed through the CLIP text encoder, producing a text embedding — a vector of numbers representing the semantic meaning of your prompt. This embedding is then fed into the diffusion model at every denoising step via a mechanism called cross-attention. At each layer of the neural network, the model attends to the text embedding, steering the denoising process toward an image that matches your description.

CLIP vs T5 and newer encoders

CLIP has a limitation: it was designed for image-text contrastive learning, not for understanding complex, compositional language. If you write "a red circle above a blue triangle," CLIP understands "red," "circle," "blue," and "triangle" but may not reliably capture their spatial relationship.

Newer models like Stable Diffusion 3 and Flux use stronger text encoders including T5-XXL, a large language model encoder. T5 understands complex syntax, relationships, and compositional descriptions much better than CLIP. This is why newer models follow complex prompts more accurately — the text understanding improved dramatically.

Samplers and Steps

When you generate an image, the model runs the reverse diffusion process for a set number of steps. Each step refines the image further. More steps generally means better quality and more detail, but diminishing returns set in quickly — going from 20 to 50 steps often barely changes the result, while taking 2.5× as long.

The sampler (also called scheduler) determines the mathematical algorithm used to update the image at each denoising step. Different samplers make different tradeoffs between speed, accuracy, and style:

  • DDIM (Denoising Diffusion Implicit Models) — One of the original samplers, good for deterministic results with fewer steps (20–50).
  • Euler and Euler a — Fast, efficient, produce good results with 20–30 steps. Euler a introduces stochastic (random) variations that can help diversity.
  • DPM++ samplers — High-quality samplers that often produce sharper, more detailed results. DPM++ 2M Karras is a popular choice for quality.
  • LCM and Lightning samplers — Distilled samplers that can produce good results in just 4–8 steps by training a special model that learns to accelerate the process.

For most practical purposes, Euler a or DPM++ 2M Karras with 20–30 steps gives excellent results. The sampler choice matters less than the model, prompt, and seed.

CFG Scale — Balancing Creativity and Prompt Adherence

The classifier-free guidance (CFG) scale (also called guidance scale) controls how strongly the model follows your text prompt versus how freely it generates. It is typically a number between 1 and 20.

  • Low CFG (1–4): The model largely ignores the prompt and produces diverse, creative, but often unprompted results.
  • Medium CFG (6–9): Good balance — the model follows the prompt reasonably while maintaining image quality. 7 is a common default.
  • High CFG (10–20): The model adheres very closely to the prompt, but images can become oversaturated, distorted, or artificially stylized.

Newer models like Flux Dev are trained differently and do not use CFG in the same way — they have simpler guidance mechanisms and generally require lower values or none at all.

What Are LoRAs?

LoRA stands for Low-Rank Adaptation. It is a technique that allows you to fine-tune a large model efficiently by training only a small set of additional parameters, rather than retraining the entire model.

In practice, a LoRA is a small file (typically 50–300 MB) that you combine with a base model to steer it toward a specific style, subject, concept, or character. The base model might be Flux Dev or SDXL, and you might load a LoRA trained on a specific artist's style, a specific person, a specific object category, or a detailed environment type.

LoRAs are extremely popular in the AI art community because:

  • They are small and easy to share
  • They can be combined — you can apply multiple LoRAs simultaneously
  • Training your own LoRA on your own photos lets you generate consistent characters or styles
  • Community repositories like Civitai host thousands of community-trained LoRAs

Related techniques include textual inversion (training a new token embedding) and DreamBooth (fine-tuning more of the model's weights for stronger personalization).

ControlNet and Guided Generation

Standard text-to-image generation is powerful but unpredictable in terms of exact composition and pose. ControlNet, introduced by Lvmin Zhang in 2023, adds precise spatial control by conditioning the generation on a reference image processed through a specific detector.

Different ControlNet models process reference images in different ways:

  • Canny: Detects edges in a reference image. The generated image follows those edges, allowing you to control the outline and structure of objects.
  • OpenPose: Detects human body keypoints (joints). Lets you control the exact pose of a generated person.
  • Depth: Estimates the depth map of a scene. Generated images maintain the same spatial arrangement and depth relationships.
  • Seg (Segmentation): Provides a semantic segmentation map showing what category of object (sky, building, person, road) occupies each region. Generated images fill in photorealistic content while maintaining the layout.
  • Scribble: Lets you draw rough sketches that become generated images.

ControlNet makes AI image generation practical for professional design workflows — you can use it to maintain a character's pose across multiple images, turn a rough sketch into a finished illustration, or generate variations of a product photo from multiple angles.

Stable Diffusion vs Flux vs Midjourney vs DALL-E — How They Compare

Tool Open Source Local Run Prompt Following Photorealism Cost
Stable Diffusion 1.x/2.x ✓ Yes ✓ Yes Moderate Good Free (local)
SDXL ✓ Yes ✓ Yes Good Very good Free (local)
Flux Dev / Flux Pro Partially ✓ Yes (Dev) Excellent Excellent Free (Dev, local)
Midjourney ★�Œ No ★�Œ No Very good Excellent $10–$60/month
DALL-E 3 ★�Œ No ★�Œ No Excellent Very good API pricing / ChatGPT Plus
Adobe Firefly ★�Œ No ★�Œ No Good Good Adobe CC subscription

Flux Dev (developed by Black Forest Labs, the team behind the original Stable Diffusion) has become a popular choice for local generation in 2026. Its T5 text encoder gives it dramatically better prompt following than older Stable Diffusion models, and it produces photorealistic images with excellent detail. The model weights are available for free for non-commercial use.

ComfyUI is the most flexible open-source interface for running Flux, SDXL, and other models locally. Its node-based workflow editor allows precise control over every aspect of the generation pipeline. It is more complex than simpler interfaces like Automatic1111 but offers considerably more power and extensibility. See the resource library for links to documentation and community resources.

Running AI Image Generation Locally vs Cloud Services

Choosing between local and cloud-based image generation involves several tradeoffs:

Local generation (ComfyUI, Automatic1111, InvokeAI)

  • Cost: No per-image cost after hardware setup. Electricity costs are minimal for occasional use.
  • Privacy: Your images and prompts never leave your machine.
  • Control: Full control over models, samplers, settings, and workflow.
  • Speed: Depends heavily on your GPU. An RTX 4090 generates a 1024×1024 Flux image in 10–20 seconds. A GTX 1080 might take several minutes.
  • Minimum GPU VRAM requirements: Flux Dev at full quality: 20+ GB VRAM. SDXL: 8–10 GB. SD 1.5: 4–6 GB. CPU generation is possible but very slow.

Cloud generation (Midjourney, DALL-E, Firefly)

  • Cost: Subscription or per-image pricing. Volume usage can become expensive.
  • Privacy: Your prompts and generated images may be used for training or logging.
  • Accessibility: No special hardware required. Works from any browser.
  • Speed: Usually 10–30 seconds per image, independent of your hardware.
  • Content policy: Cloud services have strict content filters. Local models can be run without these restrictions (at the user's responsibility).

Practical Tips for Better AI Image Generation Results

  1. Be specific, not abstract. "A photograph of a red ceramic mug on a wooden desk, morning light, soft shadows, shallow depth of field" produces more reliable results than "a nice cup of coffee."
  2. Specify the medium and style early. Words like "photograph," "oil painting," "digital illustration," or "pencil sketch" near the beginning of your prompt strongly influence the output.
  3. Use negative prompts for older models. SD 1.5 and SDXL benefit from negative prompts listing things you don't want: "blurry, watermark, text, distorted, ugly, low quality." Flux handles this better through the prompt itself.
  4. Vary the seed to explore options. The seed is a random number that determines where the generation starts. With the same prompt and seed, you always get the same image. Changing the seed explores different compositions.
  5. Use img2img for iteration. Generate a rough version, then refine it using img2img with a low denoising strength (0.4–0.6) to preserve the composition while improving details.
  6. Learn your model's vocabulary. Different models were trained on different datasets and respond differently to the same words. Experimenting with synonyms ("azure" vs "blue," "elderly" vs "aged") can unlock better results.
  7. Check the license of your model. Many checkpoints on community sites have licensing restrictions. Some are for non-commercial use only. Some prohibit specific output categories. Always check before using generated images commercially.

Licensing and AI-Generated Images

The legal status of AI-generated images is still evolving and varies by jurisdiction. Here is what is generally understood as of 2026:

  • Copyright ownership: In most jurisdictions, copyright requires human authorship. Fully AI-generated images with no human creative selection or arrangement generally cannot be copyrighted in the United States (see US Copyright Office guidance). Courts and regulators in other countries are reaching different conclusions.
  • Model licensing: The model you use may have its own license governing commercial use of generated images. For example, Flux Dev is licensed for non-commercial use only unless you use Flux Pro via the official API. SDXL's CreativeML OpenRAIL+M license permits commercial use but prohibits certain output categories.
  • Training data claims: Ongoing litigation and regulatory activity concerns whether models trained on copyrighted data infringe on the rights of original creators. This is an active legal area.
  • Practical advice: For commercial projects, use models with clear commercial licenses, check the specific license terms, document which model and version you used, and consider whether the generated images could be confused with real people or existing copyrighted characters.

For general asset licensing guidance, see the license checklist. For Creative Commons licenses that may apply to open-source model weights, see our Creative Commons guide.