🧨 ControlNet on Segment Anything 🤗
This is a demo on 🧨 ControlNet based on Meta's Segment Anything Model.
Upload an Image, Segment it with Segment Anything, write a prompt, and generate images 🤗
⌛️ It takes about 20~ seconds to generate 4 samples, to get faster results, don't forget to reduce the Nº Samples to 1.
You can obtain the Segmentation Map of any Image through this Colab:
A huge thanks goes out to @GoogleCloud, for providing us with powerful TPUs that enabled us to train this model; and to the @HuggingFace Team for organizing the sprint.
Check out our Model Card 🧨
Try some of the examples below ⬇️
👨💻 About the model
This model is based on the ControlNet Model, which allow us to generate Images using some sort of condition image. For this model, we selected the segmentation maps produced by Meta's new segmentation model called Segment Anything Model as the condition image. We then trained the model to generate images based on the structure of the segmentation maps and the text prompts given.
💾 About the dataset
For the training, we generated a segmented dataset based on the COYO-700M dataset. The dataset provided us with the images, and the text prompts. For the segmented images, we used Segment Anything Model. We then created 8k samples to train our model on, which isn't a lot, but as a team, we have been very busy with many other responsibilities and time constraints, which made it challenging to dedicate a lot of time to generating a larger dataset. Despite the constraints we faced, we have still managed to achieve some nice results 🙌
You can check the generated datasets below ⬇️