Diffusion models, the same mathematical engines powering photorealistic image generators like DALL-E and Stable Diffusion, are now being reverse-engineered to help machines understand and perceive the visual world. A comprehensive new survey published in the open-access journal Vicinagearth documents how researchers are using generative AI technology to tackle classic computer vision tasks like depth estimation, object segmentation, and surface geometry prediction, pointing toward a future where machines perceive and imagine through a single unified framework.

What Are Diffusion Models, and Why Do They Matter for Vision?

Diffusion models work by learning to reverse a gradual noising process. Instead of generating an image in one step, they start with pure random noise and iteratively refine it into coherent structure, much like gradually sharpening a blurry photograph. This approach, introduced in 2020 through Denoising Diffusion Probabilistic Models, outperformed earlier generative systems and sparked a wave of refinements.

Most modern systems don't operate on raw pixels. Instead, they compress images into a condensed mathematical space using a technique called a variational autoencoder, a strategy popularized by Latent Diffusion Models and their descendants Stable Diffusion and SDXL. This compression dramatically reduces computational cost while preserving image quality, making high-resolution generation practical on consumer hardware.

The architectural evolution has been rapid. Early diffusion systems used convolutional U-Nets, but the field has since moved to Diffusion Transformers, or DiTs, which replace the U-Net with a scalable transformer backbone. This powers flagship text-to-image systems such as PIXART-alpha, PIXART-Sigma, and the 4K-capable Wuerstchen pipeline. Newer approaches like flow matching learn a more direct trajectory from noise to image, enabling remarkably fast sampling. The latest generation, exemplified by FLUX.1 Kontext, unifies image generation and editing within a single flow matching architecture.

How Are Researchers Turning Image Generators Into Perception Engines?

The most striking finding in the survey is the reversal of direction: diffusion models are being repurposed as perception engines. Rather than classifying or regressing to predict properties, researchers now treat understanding tasks as conditional generation problems. A model called Marigold reframes monocular depth estimation, the task of inferring 3D depth from a single 2D image, as a latent diffusion task and achieves strong performance by repurposing pretrained image generators. DepthFM accelerates this approach with flow matching, while StableNormal and NormalCrafter generate temporally consistent surface normal maps from images and video.

Segmentation, the task of identifying and outlining objects in images, has followed suit. Systems like DiffusionInst and ODISE formulate instance and panoptic segmentation as conditional generation, refining masks through denoising. Training-free methods show that powerful semantic cues are already embedded in pretrained diffusion backbones, extractable through feature clustering and attention analysis without any task-specific training.

Joint frameworks like GeoWizard, Lotus, GeometryCrafter, Geo4D, and DICEPTION extend this to simultaneous prediction of depth, surface normals, camera parameters, and point clouds. Remarkably, these generative approaches often surpass discriminative specialists in zero-shot and cross-domain settings, though the multi-step denoising process still imposes a significant inference cost.

How Are Models Learning to Generate Multiple Modalities at Once?

The survey documents a shift toward flexible latent frameworks that can mask, fuse, and weigh heterogeneous signals spanning appearance, geometry, semantics, and motion. Two architectural strategies dominate this emerging landscape:

• Multi-modal VAE approaches: Systems like Orchid and Trellis encode RGB images, depth maps, and surface normals, or point clouds and semantic features, into a shared latent space decoded into different modality-specific outputs, enabling coherent joint synthesis for 3D asset creation and sensor fusion.
• Shared VAE approaches: Systems like Matrix3D perform pose estimation, depth prediction, and novel view synthesis in one masked multi-modal diffusion transformer, while VideoJAM jointly models appearance and optical flow with an inner-guidance mechanism for temporally coherent motion.
• World-consistent synthesis: Models like Voyager synthesize aligned RGB and depth video for unbounded, world-consistent 3D scene expansion, and JointDiT captures the joint RGB-depth distribution through adaptive weighting of different modalities.

What About Video Generation and Temporal Consistency?

Video generation has followed a parallel but more demanding trajectory, because adding a temporal dimension multiplies computational complexity and introduces the problem of temporal coherence. Early systems such as Video Diffusion Models extended 2D U-Nets to 3D by inserting temporal attention layers, while Make-A-Video, Imagen Video, and MagicVideo demonstrated that large-scale image priors could be transferred to video with relatively little paired text-video data.

Latent-space approaches like LVDM compressed video into low-dimensional representations for efficient long-form generation, and Stable Video Diffusion scaled latent video diffusion to large datasets by building on pretrained image models. More recently, transformer-based architectures including CogVideoX, LATTE, and OpenAI's Sora have adopted the Diffusion Transformer to better fuse linguistic semantics with long-range motion, while industrial-scale systems such as HunyuanVideo, Kling, Mochi, Wan, and SkyReels-V1 continue to push quality and generalization boundaries.

How Are Creators Gaining Precise Control Over Generated Content?

Text prompts alone proved too blunt an instrument for creators who need precise control over spatial layout, human pose, or camera motion. Researchers answered with structured conditioning techniques. ControlNet, perhaps the most influential contribution, attaches lightweight zero-initialized branches to a frozen diffusion model, translating depth maps, edges, poses, and segmentation masks into feature-space guidance without disturbing the base model's knowledge.

Video systems like Gen-1 decoupled structure from content, using monocular depth as a motion scaffold and CLIP embeddings as appearance signals. A second wave of 3D-aware methods goes further, injecting camera trajectories as ray embeddings or generating coarse point clouds as geometric scaffolds before refinement. Systems such as CameraCtrl, ViewCrafter, GEN3C, and Uni3C enable explicit viewpoint control and novel view synthesis, opening applications in filmmaking, simulation, and robotics.

Steps to Understanding the Unified Vision-Generation Framework

• Recognize the convergence: The same diffusion model architecture now powers both image generation and perception tasks like depth estimation and object segmentation, suggesting machines may soon perceive and imagine through one shared framework.
• Understand the efficiency gains: Latent-space compression using variational autoencoders dramatically reduces computational cost while preserving fidelity, making high-resolution generation and perception practical on consumer hardware rather than requiring massive data centers.
• Appreciate the control mechanisms: Structured conditioning techniques like ControlNet allow creators to guide generation with precise spatial information, depth maps, and camera trajectories, moving beyond simple text prompts to enable professional-grade creative control.
• Track the multi-modal frontier: Emerging systems that jointly generate and understand multiple modalities, such as RGB images paired with depth maps and semantic segmentation, signal a shift toward flexible frameworks that can fuse heterogeneous signals for 3D asset creation and sensor fusion applications.

What Does This Mean for the Future of AI Vision?

The survey's most ambitious finding concerns unified models that generate and understand in a single system. The authors trace a lineage from masked-autoencoder and tokenized approaches such as MultiMAE, UNIFIED-IO and its sequel, the 4M family, Sapiens, and PixelWorld, which consolidate dozens of tasks and modalities into shared token spaces. Diffusion-based successors now close the loop. UniReal treats image tasks as discontinuous video frames, OneDiff frames tasks as views at different noise levels, and MMGen and OmniGen2 unify generation, editing, and understanding with parallel transformers. On the video side, OmniVDiff jointly models RGB, depth, segmentation, and edges through a shared 3D VAE.

This convergence represents a fundamental shift in how machines approach visual intelligence. Rather than building separate systems for perception and generation, researchers are discovering that the same mathematical machinery can handle both tasks. The implications extend beyond efficiency; they suggest that future AI systems may learn to see and imagine in ways that more closely mirror human cognition, where understanding and creativity are deeply intertwined rather than isolated capabilities.