Use ControlNet to condition diffusion on pose/edge maps and IP‑Adapter to inject identity features, then fuse their embeddings in the Stable Diffusion pipeline.
Step‑by‑step implementation
1. Data preparation
- Gather 3–5 reference portraits of the brand character.
- Run OpenPose (or MMPose) to obtain a 512×512 pose map.
- Compute CLIP image embeddings with the IP‑Adapter encoder: ip_encoder.encode(image).
2. Environment
```python
pip install "diffusers[torch]" controlnet_aux ip_adapter tqdm
```
3. Model loading
```python
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from ip_adapter import IPAdapter
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/sd-controlnet-openpose", torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
controlnet=controlnet,
torch_dtype=torch.float16
).to("cuda")
ip_adapter = IPAdapter(pipe, "h94/IP-Adapter", torch_dtype=torch.float16)
```
4. Configuration
- controlnet_conditioning_scale = 1.0
- ip_adapter_weight = 0.8
- guidance_scale = 7.5
- num_inference_steps = 50
- Set a fixed generator = torch.Generator("cuda").manual_seed(42) for reproducibility.
5. Generation
```python
pose = Image.open("pose.png").convert("RGB")
prompt = "portrait of a futuristic brand mascot, cinematic lighting"
image = pipe(
prompt,
image=pose,
controlnet_conditioning_scale=1.0,
ip_adapter_weight=0.8,
guidance_scale=7.5,
num_inference_steps=50,
generator=generator,
).images[0]
image.save("output.png")
```
ControlNet vs IP‑Adapter (quick reference)
| Feature | ControlNet | IP‑Adapter |
|------------------|--------------------------------|--------------------------------|
| Conditioning type| Spatial maps (pose, edge, depth) | Identity embedding (CLIP) |
| Typical scale | 0.5–1.5 | 0.5–1.0 |
| Best for | Pose fidelity, layout control | Consistent character appearance|
Gotcha: If the pose map resolution differs from the diffusion model’s latent size (e.g., 640×640 vs 512×512), the character’s limbs will drift; always resize the control map to the pipeline’s height/width before feeding it in.