Diffusion Model Paper Reading — LoRA in Diffusion Models + Fine-Tuning Diffusion Model For Real Client
TL;DR In this session, we’ll break down the paper “A Closer Look at Parameter-Efficient Tuning in Diffusion Models” and then do a practical lab: fine-tuning the Qwen-Image-Edit model for a real client image-editing workflow. You’ll leave with a clear mental model of PEFT for diffusion (adapters/LoRA-style modules, where to insert them, why it matters) + a concrete training recipe you can reuse.
This is part of our ongoing Diffusion Model Paper Reading Group — a friendly, online community across NY, SF, Toronto, and Boston, open to anyone curious about modern generative models.
👌 Learning requirements You’ll be fine as long as you’re:
Curious about customizing diffusion/image-editing models without full fine-tuning
Comfortable skimming a technical paper + discussing design tradeoffs
Open to light math + training intuition (hands-on code is optional but encouraged)
🗓 Session format (suggested 2 hours)
~30 min — Paper walkthrough (PEFT design space + what actually matters)
Why full fine-tuning is expensive for diffusion customization
Adapter design space (where to insert modules, what to tune)
Key takeaway: insertion position matters a lot (especially around cross-attention) + a practical recipe that can match full fine-tuning with ~<1% extra parameters (paper reports ~0.75%). oai_citation:0‡arXiv
~30 min — Hands-On Lab: Fine-tune Qwen-Image-Edit for a real client
Problem framing: “image edit requests” → outputs that match client style/constraints
Dataset format: (input image, instruction prompt, target edit) pairs
Training approach: parameter-efficient fine-tuning (LoRA/adapters) so you can iterate fast
How Qwen-Image-Edit works at a high level (semantic + appearance control) and what that implies for fine-tuning oai_citation:1‡Hugging Face
Wrap-up: evaluation checklist + how to ship a usable workflow (not just a demo)
📚 Pre-class learning (optional but recommended) Paper:
A Closer Look at Parameter-Efficient Tuning in Diffusion Models — arXiv:2303.18181 https://arxiv.org/abs/2303.18181 oai_citation:2‡arXiv
Optional code reference (paper authors’ repo):
Official/fixed code repo: https://github.com/Xiang-cd/unet-finetune oai_citation:3‡GitHub
Qwen-Image-Edit background (optional):
Model card: https://huggingface.co/Qwen/Qwen-Image-Edit oai_citation:4‡Hugging Face
Workflow example (ComfyUI docs): https://docs.comfy.org/tutorials/image/qwen/qwen-image-edit oai_citation:5‡ComfyUI
💻 What to bring (for the lab)
A laptop is enough to follow along conceptually
If you want to actually run training live: access to a GPU (local or cloud/Colab)
We’ll provide a minimal training template + dataset schema during the session
👥 Who this is for
Engineers / builders who want to customize image generation or image editing models
People exploring real-world diffusion fine-tuning (style, product images, brand constraints)
Anyone who wants a “paper → implementation → real use case” loop
🧠 About the Diffusion Model Reading Group A peer-led learning journey for engineers, students, researchers, and builders exploring diffusion architectures and modern generative AI.
No ML background required — just curiosity and comfort with technical ideas
2–4 hours/week if you follow along with paper reading + optional projects
Supportive community of people already working in or entering the AI industry