Cones: Concept Neurons in Diffusion Models for Customized Generation

by   Zhiheng Liu, et al.

Human brains respond to semantic features of presented stimuli with different neurons. It is then curious whether modern deep neural networks admit a similar behavior pattern. Specifically, this paper finds a small cluster of neurons in a diffusion model corresponding to a particular subject. We call those neurons the concept neurons. They can be identified by statistics of network gradients to a stimulation connected with the given subject. The concept neurons demonstrate magnetic properties in interpreting and manipulating generation results. Shutting them can directly yield the related subject contextualized in different scenes. Concatenating multiple clusters of concept neurons can vividly generate all related concepts in a single image. A few steps of further fine-tuning can enhance the multi-concept capability, which may be the first to manage to generate up to four different subjects in a single image. For large-scale applications, the concept neurons are environmentally friendly as we only need to store a sparse cluster of int index instead of dense float32 values of the parameters, which reduces storage consumption by 90% compared with previous subject-driven generation methods. Extensive qualitative and quantitative studies on diverse scenarios show the superiority of our method in interpreting and manipulating diffusion models.


page 1

page 7

page 8

page 13

page 14

page 15

page 16

page 17


Disentangling Neuron Representations with Concept Vectors

Mechanistic interpretability aims to understand how models store represe...

Subject-driven Text-to-Image Generation via Apprenticeship Learning

Recent text-to-image generation models like DreamBooth have made remarka...

SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

Diffusion models have achieved remarkable success in text-to-image gener...

NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networks

Existing research on making sense of deep neural networks often focuses ...

Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models

Public large-scale text-to-image diffusion models, such as Stable Diffus...

Dynamic-structured Semantic Propagation Network

Semantic concept hierarchy is still under-explored for semantic segmenta...

Concept-Monitor: Understanding DNN training through individual neurons

In this work, we propose a general framework called Concept-Monitor to h...

Please sign up or login with your details

Forgot password? Click here to reset