What is Stable Diffusion?

Stable Diffusion is a text-to-image model that transforms a text prompt into a high-resolution image. For example, if you type in a cute and adorable bunny, Stable Diffusion generates high-resolution images depicting that — a cute and adorable bunny — in a few seconds. Click “Select another prompt” in Diffusion Explainer to change prompts and check the fascinating images generated from each prompt!

How does Stable Diffusion work?

Stable Diffusion first changes the text prompt into a text representation, numerical values that summarize the prompt. The text representation is used to generate an image representation, which summarizes an image depicted in the text prompt. This image representation is then upscaled into a high-resolution image.

You may wonder why Stable Diffusion introduces image representation instead of directly generating high-resolution images. The reason is computational efficiency. Doing most computations on compact image representation instead of a high-resolution image significantly reduces the time and cost for the computations while maintaining high image quality.

The image representation, which starts as a random noise, is refined over multiple timesteps to reach the image representation for a high-quality image with strong adherence to the text prompt. The number of refining timesteps is typically set as 50 or 100; we fix it to 50 in Diffusion Explainer.

We break down the image generation process of Stable Diffusion into three main steps:

1. Text Representation Generation: Stable Diffusion converts a text prompt into a text vector representation.
2. Image Representation Refining: Starting with random noise, Stable Diffusion refines the image representation little by little, with the guidance of the text representation. Stable Diffusion repeats the refining over multiple timesteps (50 in our Diffusion Explainer).
3. Image Upscaling: Stable Diffusion upscales the image representation into a high-resolution image.

Now, let's look closer into each process.

Text Representation Generation

Clicking Text Representation Generation shows how a text prompt is converted into a text representation, a vector that summarizes the prompt. It consists of two steps: tokenizing and text encoding.

1. Tokenizing

Tokenizing is a common way to handle text data to change the text into numbers and process them with neural networks.

Stable Diffusion tokenizes a text prompt into a sequence of tokens. For example, it splits the text prompt a cute and adorable bunny into the tokens a, cute, and, adorable, and bunny. Also, to mark the beginning and end of the prompt, Stable Diffusion adds <start> and <end> tokens at the beginning and the end of the tokens. The resulting token sequence for the above example would be <start>, a, cute, and, adorable, bunny, and <end>.

For easier computation, Stable Diffusion keeps the token sequences of any text prompts to have the same length of 77 by padding or truncating. If the input prompt has fewer than 77 tokens, <end> tokens are added to the end of the sequence until it reaches 77 tokens. If the input prompt has more than 77 tokens, the first 77 tokens are retained and the rest are truncated. The length of 77 was set to balance performance and computational efficiency.

2. Text encoding

Stable Diffusion converts the token sequence into a text representation. To use the text representation for guiding image generation, Stable Diffusion ensures that the text representation contains the information related to the image depicted in the prompt. This is done by using a special neural network called CLIP.

CLIP, which consists of an image encoder and a text encoder, is trained to encode an image and its text description into vectors that are similar to each other. Therefore, the text representation for a prompt computed by CLIP’s text encoder is likely to contain information about the images described in the prompt. You can display the visual explanations by clicking the Text Encoder above.

Image Representation Refining

Stable Diffusion generates image representation, a vector that numerically summarizes a high-resolution image depicted in the text prompt. This is done by refining a randomly initialized noise over multiple timesteps to gradually improve the image quality and adherence to the prompt. You can change the initial random noise by adjusting the seed in Diffusion Explainer. Click Image Representation Refiner to visualize each refinement step, which involves noise prediction and removal.

1. Noise Prediction

At each timestep, a neural network called UNet predicts noise in the image representation of the current timestep. UNet takes three inputs:

1. Image representation of the current timestep
2. Text representation of the prompt to guide what noise should be removed from the current image representation to generate an image adhering to the text prompt
3. Timestep to indicate the amount of noise remaining in the current image representation

In other words, UNet predicts a prompt-conditioned noise in the current image representation under the guidance of the text prompt's representation and timestep.

However, even though we condition the noise prediction with the text prompt, the generated image representation usually does not adhere strongly enough to the text prompt. To improve the adherence, Stable Diffusion measures the impact of the prompt by additionally predicting generic noise conditioned on an empty prompt (" ") and subtracting it from the prompt-conditioned noise:

impact of prompt = prompt-conditioned noise - generic noise

In other words, the generic noise contributes to better image quality, while the impact of the prompt contributes to the adherence to the prompt. The final noise is a weighted sum of them controlled by a value called guidance scale:

generic noise + guidance scale x impact of prompt

A guidance scale of 0 means no adherence to the text prompt, while a guidance scale of 1 means using the original prompt-conditioned noise. Larger guidance scales result in stronger adherence to the text prompt, while too large values can lower the image quality. Change the guidance scale value in Diffusion Explainer and see how it changes the generated images.

2. Noise Removal

Stable Diffusion then decides how much of the predicted noise to actually remove from the image, as determined by an algorithm called scheduler. Removing small amounts of noise helps refine the image gradually and produce sharper images.

The scheduler makes this decision by accounting for the total number of timesteps. The downscaled noise is then subtracted from the image representation of the current timestep to obtain the refined representation, which becomes the image representation of the next timestep:

image representation of timestep t+1 = image representation of timestep t - downscaled noise

Image Upscaling

After all denoising steps have been completed, Stable Diffusion uses a neural network called Decoder to upscale the image representation into a high-resolution image. The refined image representation fully denoised with the guidance of the text representations would result in a high-resolution image strongly adhering to the text prompt.

How do prompt keywords affect image generation?

Writing text prompts can be very heuristic and repetitive. For example, starting from the prompt a cute bunny, you should repetitively add and remove keywords such as in the style of cute pixar character, until you reach to the desired image.

Therefore, understanding how prompt keywords affect image generation would be greatly helpful for writing and refining your prompt. Click the keywords highlighted in the text prompt and compare the image generation of the two prompts that differ only in the keywords.

What can we change?

You have control over text prompt and hyperparameters in our Diffusion Explainer to change the generated images:

• Text prompt: Description of the image you want to generate. A more detailed text prompt generally leads to better quality images.
• Seed: Random seed for the initialization of the image representation at timestep 0. Changing the seed will result in different image representation at timestep 0 and therefore different images.
• Guidance scale: How closely the generated image adheres to the text prompt. Increasing the guidance scale leads to stronger adherence but may make the generated images overly exaggerated.

Additionally, there are other hyperparameters that are not included in the Diffusion Explainer, such as the total number of timesteps, image size, and the type of scheduler.

How is Diffusion Explainer implemented?

We implement the interactive visualizations for Diffusion Explainer using Javascript and D3.js.

Who developed the Diffusion Explainer?

Diffusion Explainer was developed by Seongmin Lee, Ben Hoover, Hendrik Strobelt, Jay Wang, Anthony Peng, Austin Wright, Kevin Li, Haekyu Park, Alex Yang, and Polo Chau.