welcome to The Prompter #003!
this is a weekly newsletter written by krea.ai for AI whisperers and prompters.
📰 AI news
stability AI keeps its momentum
the release of Stable Diffusion has been flawless, more than 15,000 people are generating around 2 million images everyday.
after reading the Stable Diffusion launch announcement, we were amazed at how Stability managed to get the model working in less than 10GB of VRAM.
well, Emad Mostaque, the CEO of Stability, shared the news that they managed to get Stable Diffusion working on just 5.1 Gb VRAM!
this means that soon anyone will be able to access and use the model through Google Colab or their own GPUs, and generate images in seconds for free.
in a recent interview by Yannic Kilcher, Emad shares more about their plans for democratizing AI.
the code for Stable Diffusion is out.
the code was released in GitHub repository from CompVis, the research group behind Sable Diffusion.
there, we saw that they had implementations for more use cases than just text to image, such as image editing!
we had the chance to play with this new functionality, the results are pretty cool.
the way how this feature works is by generating on top of a starting image (in this case the one on the left) from a text prompt, and we are be able to regulate ho much change we want from the original image.
this is very similar to how GauGAN works, but since Stable Diffusion has a wider variety of knowledge, it provide us with endless creative possibilities.
this is just the beginning of what having access to the weights of this model means, soon we will see amazing applications and creative uses for this model.
first attempts at training new Stable Diffusion models
since the code is out, the only thing needed to create a new variant of Stable Diffusion is a bunch of GPUs and a large dataset of images with captions.
Justin Pinkey shared his latest experiments doing exactly that.
his results are pretty impressive, diffusion models are very hard to train and having these results in just one night is a very good sign for anyone planning to train its own model.
🛠️ tools for prompting
get prompts from images with the CLIP Interrogator
@pharmapsychotic released a new Colab Notebook that, given an image, is capable of creating a text prompt representing its content as style.
see how the model is able to create the prompt “a man standing on top of a bridge over a city, cyberpunk art by Vincent Lefevre, behance contest winner, altermodern, cityscape, synthwave, matte painting” from just a single image.
the final prompt includes a description of the content in the image, artists related with the aesthetic of it, style modifiers, artistic mediums, and even art movements.
the way how it works is very simple: it uses BLIP to generate a description of the content in the image, and then uses CLIP to match the image with stylistic modifiers from four large lists of artists, flavors, art movements, and mediums.
in this github repository you can see the code and the data that he used.
stable diffusion artist studies
Stable Diffusion Artist Studies has over 600 artists now.
this is the the result of the hard work of @proximasan, @EErratica, @KyrickYoung, and @sureailabs, that spent countless hours searching for artists and exploring how AI models understand and represent them.
in each item from the studies, they share three generations created with “a portrait of a character in a scenic environment by [artist]” and three resulting from “a building in a stunning landscape by [artist]”.
you should check out their parrot zone, they have a work-in-progress studies around Stable Diffusion Modifiers, a HUGE Disco Diffusion Artist Studies, a Stable Diffusion Seed Bank where anyone can contribute with cool prompts and seeds, and even some Colab Notebooks by @KyrickYoung to train your own text generator model, or to use FILM to interpolate images.
DALL-E Prompt Helper
this is a very neat tool that didn’t have the attention it deserves.
it consist of a chrome extension that changes the way how the DALL-E interface looks, adding new features like the prompt helper, auto-completion with GPT-3, and the possibility of downloading a whole batch of images.
their dark mode makes DALL-E look very sexy!
follow @altryne for more.
🎨 AI Art
runwayML, videos with Stable Diffusion
infinite zooms
@KyrickYoung messing around with rotation effects
Xander’s masterpiece using Stable Diffusion interpolations
🦾 krea updates
since the release of Stable Diffusion we have not been sleeping much…
we created a large knowledge base of text prompts for Stable Diffusion.
we designed a new interface for prompting that makes it seamless to visualize and create complex text prompts in seconds.
we implemented the new interface and started to share it with beta testers.
here’s the video we shared our early users explaining how it works: VIDEO
we’d love to hear your thoughts, ideas, and feedback, reach out!