ComfyUI ControlNet and Upscaling Guide

Latent Vision

Introduction [00:00]

• This video is the third installment in the ComfyUI Advanced Understanding series, which aims to explain how Stable Diffusion and ComfyUI function.
• The main topics for this episode are ControlNet and Upscaling.
• Viewers are encouraged to watch parts one [Image 2] and two [Image 4] of the series if they haven't already.

It's just noise [00:49]

• Understanding Latent Noise Generation [00:49]
  • The video begins with a KSampler Advanced workflow generating Christmas decorations [Image 0].
  • By setting "return_with_leftover_noise" to "enable" and "end_at_step" to 1, the presenter shows the initial noisy image before any denoising steps [Image 6].
  • This initial noisy image contains all the information needed to generate the final picture.
  • As "end_at_step" is increased, the image slowly forms from the noise.
• Influencing Noise Generation [01:36]
  • Instead of starting with an empty latent image, a pre-generated "scribble" image (a black canvas with two red circles of noise) is loaded and encoded into latent space.
  • When generation starts with this pre-defined noise, the output images consistently feature two red Christmas ornaments, demonstrating how initial noise influences composition [Image 9].
  • This demonstrates that the initial noise heavily influences the final composition.
• Using Noise Masks [02:52]
  • A grayscale image with a black center and white border is used as a mask.
  • The mask is loaded and applied using "Set Latent Noise Mask" node, influencing noise generation in specific areas (e.g., creating a frame).
  • This technique allows for driving composition in defined regions of the image.

ControlNet Canny [03:40]

• Introduction to ControlNet Nodes [03:40]
  • ControlNets are introduced as tools to manipulate noise during the entire sampling process.
  • ComfyUI primarily offers two ControlNet nodes: Apply ControlNet and Apply ControlNet (Advanced).
  • The advanced version is recommended as it offers more control, allowing manipulation of positive, negative, and image conditioning, strength, and time-stepping (start_percent, end_percent).
• Implementing Canny ControlNet for Pose Control [04:23]
  • An anime girl is generated with blue hair [Image 11]. To guide her pose, a reference image of a woman in a specific pose is loaded.
  • A Canny Edge Preprocessor is used to extract edges from the reference image, making it understandable for ControlNet.
  • Thresholds (low_threshold, high_threshold) in Canny adjust the detail level of the extracted edges.
  • The Canny preprocessed image is connected to the image input of the Apply ControlNet (Advanced) node [Image 12].
  • A Load ControlNet Model node is used to select the control_v11p_sd15_canny.safetensors model.
  • The ControlNet is connected to the KSampler's positive and negative conditioning.
• Fine-tuning ControlNet Influence [05:40]
  • A strength of 1.0 is often too high, as it gives the ControlNet too much control.
  • Reducing strength to 0.7 allows the base model (checkpoint) more creative freedom while still maintaining the pose guidance.
  • Generated variations from Canny ControlNet are shown [Image 13].
  • start_percent and end_percent control the duration of ControlNet's influence during sampling.
  • Example: Setting strength to 1.0 from 0% to 30% of steps, then 0% strength for the rest, ensures initial composition is strong, while allowing the model freedom later.
  • Using stochastic samplers like dpmpp_2m_sde or euler_ancestral is beneficial when start_percent is greater than 0, as they offer more freedom in noise movement between steps.

Tidy the mess [08:56]

• Workflow Organization Best Practices [08:56]
  • The presenter emphasizes the importance of keeping ComfyUI workflows organized for readability and ease of understanding, even if he doesn't always follow this advice.
  • Avoid overlapping nodes and connections: Keep pipelines clear so all connections are visible without moving nodes around.
  • Use reroutes: Utilize reroute nodes to manage complex connections and prevent spaghetti-like graphs.

ControlNet OpenPose [10:33]

• Introduction to OpenPose [10:33]
  • OpenPose is introduced as another ControlNet technique, specializing in estimating skeletal poses from images.
  • The DWPose Estimator node is used as the preprocessor.
  • It can detect hands, bodies, and faces, which can be enabled or disabled based on specific needs.
  • The processed pose keypoints (stick figure) are then fed into the ControlNet.
• Full Body Pose Control [11:50]
  • A full-body reference image of a woman in a pink dress is used.
  • The latent image dimensions are adjusted to match the aspect ratio of the full-body image (e.g., 512x768).
  • The DWPose Estimator is configured to detect body, hands, and face.
  • A Realistic Vision checkpoint is used for the base model, and the prompt is adjusted to "beautiful blonde woman standing wearing a fashion pink dress".
  • The generated image successfully matches the pose of the reference image, along with the OpenPose workflow [Image 14].
  • The IPAdapter (Image Prompt Adapter) unified loader is used with a "STANDARD" model strength [0] to transfer the style of the pink dress to the generated image.

ControlNet Tile [14:09]

• Tile ControlNet for Color Transfer [14:09]
  • Tile ControlNet is explained as a color transfer mechanism, directly using pixel values without a preprocessor.
  • The control_v11f1e_sd15_tile.safetensors model is used.
  • A famous painting (Mona Lisa) is used as a reference image for the Tile ControlNet.
  • The prompt describes a cyberpunk scene.
  • The Tile ControlNet helps apply the colors and tones of the Mona Lisa to the cyberpunk image, resulting in a loose stylistic resemblance [Image 16].
  • Adjusting the end_percent (e.g., to 0.8) for Tile ControlNet can increase its influence throughout the generation process, making the output more closely resemble the reference colors.
  • For stronger resemblance, the latent image can be encoded from the reference image, ensuring initial noise is derived from the Mona Lisa.

Simple Upscale [15:29]

• Upscaling Options [15:29]
  • Upscaling can be done in two ways:
    1. Pixel Space: Using Upscale Image By node, applying standard interpolation methods like Lanczos or Bicubic.
    2. Latent Space: Using Upscale Latent By node, which operates on the compressed latent representation of the image.
  • A 2x upscale is demonstrated on a generated cyberpunk warrior image [Image 16].
  • Upscaling methods like Lanczos or Bicubic are compared.
  • Often, simply upscaling can introduce small glitches or maintain imperfections due to the model not being trained on very high resolutions.

Noise injection [18:11]

• Addressing Upscaling Artifacts with Noise Injection [18:11]
  • When upscaling, if the base model isn't trained for higher resolutions, it might struggle to generate fine details, leading to glitches.
  • To fix this, noise can be injected back into the upscaled image, giving the model a chance to "re-render" details.
  • The Inject Noise node is used, taking the upscaled latent image and a noisy latent image (from Noisy Latent Image node) as inputs, as shown in the workflow [Image 18].
  • An initial image of a cyberpunk warrior with some noise is shown [Image 17, Image 19].
  • The strength parameter of Inject Noise controls how much new noise is added, influencing the level of detail generation.
  • The start_at_step and end_at_step parameters of the KSampler determine when the noise injection occurs.
  • The start_at_step is set to the point where the image was last stable before upscaling (e.g., 12 steps if initial generation was 25 steps).
  • A stochastic sampler (e.g., dpmpp_2m_sde) is often recommended for noise injection to provide the model more freedom to add details, leading to a more refined image [Image 25].

Upscale with tiling [22:18]

• Overcoming VRAM Limitations and Enhancing Details with Tiling [22:18]
  • Tiling is a technique to subdivide a large image into smaller chunks, process each chunk (e.g., upscale), and then stitch them back together.
  • This is crucial for handling large images that exceed VRAM capacity and for applying consistent details across an image.
  • The Image Tile node is used to split the image into a batch of smaller images (tiles). Parameters include cols, rows, and overlap.
  • Image From Batch nodes are used to extract individual tiles for processing.
  • Padding (e.g., 25% overlap) is used during tiling to ensure smooth transitions between stitched tiles and avoid sharp seams.
• Combining Multiple ControlNets with Tiling [23:33]
  • The video demonstrates combining Depth ControlNet (using Depth Anything V2) and Tile ControlNet with tiling, shown in the complex workflow [Image 26].
  • Depth maps provide structural guidance (volumes), while Tile ControlNet assists with color and texture.
  • Each tile is processed by both ControlNets individually.
  • The processed tiles are then reassembled using Image Composite Masked nodes.
  • For seamless stitching, Solid Mask and Feather Mask nodes are used to create soft edges between the tiles, blending them smoothly.

Upscale with model [30:28]

• Using Dedicated Upscaling Models [30:28]
  • Dedicated upscaling models, like those found on openmodeldb.info, are specifically trained to enhance image resolution and detail.
  • The Upscale Image (using Model) node is used to apply these models, as seen in the workflow [Image 32].
  • An Upscale Model Loader node loads the desired upscaling model (e.g., 4x_UltraSharp.pth).
  • The model scales the image up (e.g., 4x) to a high resolution (e.g., 2048x2048).
  • After initial upscaling, a second KSampler pass with noise injection can be applied to add further details and refine the image, especially if the original image was lower resolution.
• Color Correction Post-Upscale [31:49]
  • Upscaling or noise injection can sometimes alter the original colors.
  • The Image Apply LUT node (Look-Up Table) can be used to apply color grading or correction to match desired color profiles [Image 32].
  • LUT files (.cube files) can be loaded and applied with adjustable strength to fine-tune the color output, resulting in a detailed image [Image 31].
• Advanced Upscaling Workflow (Convoluted Example) [32:18]
  • The video showcases a very complex workflow combining multiple tiling steps, noise injection, and different ControlNets to achieve extremely high-resolution and detailed images from an SD1.5 checkpoint [Image 30].
  • This elaborate setup demonstrates the extensive control available in ComfyUI when using these techniques.
  • The presenter emphasizes that while complex, understanding the underlying technology makes it manageable.
  • Tools like ComfyUI_UltimateSDUpscale or Tiled Diffusion can automate many of these steps for daily use.

Outro [33:55]

• The presenter concludes by hoping viewers found the third installment of the series helpful in demystifying upscalers, detailers, and segmentators, ending with a final high-resolution image [Image 33].