Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models

by   Chen Henry Wu, et al.

Generative models (e.g., GANs and diffusion models) learn the underlying data distribution in an unsupervised manner. However, many applications of interest require sampling from a specific region of the generative model's output space or evenly over a range of characteristics. To allow efficient sampling in these scenarios, we propose Generative Visual Prompt (PromptGen), a framework for distributional control over pre-trained generative models by incorporating knowledge of arbitrary off-the-shelf models. PromptGen defines control as an energy-based model (EBM) and samples images in a feed-forward manner by approximating the EBM with invertible neural networks, avoiding optimization at inference. We demonstrate how PromptGen can control several generative models (e.g., StyleGAN2, StyleNeRF, diffusion autoencoder, and NVAE) using various off-the-shelf models: (1) with the CLIP model, PromptGen can sample images guided by text, (2) with image classifiers, PromptGen can de-bias generative models across a set of attributes, and (3) with inverse graphics models, PromptGen can sample images of the same identity in different poses. (4) Finally, PromptGen reveals that the CLIP model shows "reporting bias" when used as control, and PromptGen can further de-bias this controlled distribution in an iterative manner. Our code is available at


page 8

page 22

page 23

page 24

page 25

page 26

page 27

page 28


Exponential Tilting of Generative Models: Improving Sample Quality by Training and Sampling from Latent Energy

In this paper, we present a general method that can improve the sample q...

Divide and Compose with Score Based Generative Models

While score based generative models, or diffusion models, have found suc...

Minimizing Trajectory Curvature of ODE-based Generative Models

Recent ODE/SDE-based generative models, such as diffusion models, rectif...

disco: a toolkit for Distributional Control of Generative Models

Pre-trained language models and other generative models have revolutioni...

TR0N: Translator Networks for 0-Shot Plug-and-Play Conditional Generation

We propose TR0N, a highly general framework to turn pre-trained uncondit...

Preconditioned Score-based Generative Models

Score-based generative models (SGMs) have recently emerged as a promisin...

Fingerprints of Generative Models in the Frequency Domain

It is verified in existing works that CNN-based generative models leave ...

Please sign up or login with your details

Forgot password? Click here to reset