Answer: CLIP‑based embeddings are produced by a frozen dual‑encoder that maps images and text into a shared latent space for similarity scoring, whereas generative multimodal models use an encoder‑decoder diffusion or autoregressive backbone that jointly learns to reconstruct or generate images conditioned on text.
Architectural comparison
1. Core modules
- CLIP: Image encoder (ViT‑B/32, ViT‑L/14) + Text encoder (Transformer). Both output a 512‑ or 768‑dim vector; training uses contrastive loss (InfoNCE).
- Generative multimodal (e.g., Stable Diffusion 2.1): Text encoder (CLIP‑ViT‑L/14), Latent UNet diffusion model, VAE decoder, optional safety checker.
2. Training objective
- CLIP: L = -log[ exp(sim(i,t)/τ) / Σ_j exp(sim(i,t_j)/τ) ].
- Diffusion: Denoising score matching across 1000 timesteps, plus classifier‑free guidance.
3. Inference workflow
- CLIP: image_feat = model.encode_image(img); text_feat = model.encode_text(txt); score = cosine_similarity(image_feat, text_feat).
- Stable Diffusion (Python):
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1")
image = pipe("a photorealistic cat", num_inference_steps=50, guidance_scale=7.5).images[0]Decision checklist
- Need fast similarity lookup → CLIP.
- Need controllable image synthesis → Diffusion.
- Memory budget <2 GB per batch → ViT‑B/32 CLIP.
- Accept latency ~2 s per generation → Diffusion with 50 steps.
Gotcha: When swapping the CLIP text encoder inside a diffusion pipeline, the embedding dimension must match the UNet’s cross‑attention projection (usually 768); mismatched dimensions cause a runtime error that’s silent in lazy loading frameworks.