welcome to The Prompter #006!
this is a weekly newsletter written by krea.ai to keep you updated with the latest news around prompt engineering.
📰 AI news
image outpainting in DALL·E
OpenAI announced the release of a new feature for editing images in the DALL·E playground with outpainting.
outpainting is similar to the previous image editing (where we could create a mask to decide where we wanted to apply the generation), with the difference that now we can also generate content outside the image in an infinite canvas.
folks were already outpainting images manually or with their custom scripts (@deksi even shared a tutorial about it!), but this new UI makes it so much easier.
Stable Diffusion 1.5 beta is out
just a week after Stability started beta testing the 1.5 version of Stable Diffusion, the new model is up and running in Dream Studio—kudos to the Stability team!
this version is also available to use through their API and it has the same cost as the previous one.
the new model consists of a fine-tuned version of their previous model, so we can expect better quality without affecting speed.
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
this work (by Google AI) is capable of expanding the vocabulary of an Imagen model with “words” that represent images.
the way how text-to-image models understand text, is by learning a vocabulary that encodes the semantics of words (or parts of words).
in this work, they are able to “teach” Imagen a new word that can perfectly represent the content of a set of images.
they can do this with just 2-5 input images and a concept that describes the image (like “dog”).
once the new word is learned, it can be used in combination with regular text prompts and re-create whatever we had in the input images with any modification we want.
this approach is similar to the good “old” textual inversions with the difference that the results here are not just a new textual representation of the images, but whole new fine-tuned models—since both the encoder-decoder and the super-resolution modules from Imagen are re-trained while learnind the new “word".
[announcement tweet][dreambooth website]
image variations with Stable Diffusion
Justin Pinkney released his code for creating image variations with Stable Diffusion.
this was one of the nice things that we were only able to do with DALL·E, where we could produce generations that preserved the style and content of an input image.
his method consists of substituting the textual encoding from Stable Diffusion with CLIP image embeddings, and re-train the model using images as input instead of text.
really neat!
[announcement tweet][gradio demo]
custom Stable Diffusion models
Justin has also been showcasing some of his ongoing experiments with custom Stable Diffusion models.
it seems like he is fine-tuning the model with a Pokemon dataset!
🛠️ tools for prompting
AUTOMATIC1111 created the most complete Stable Diffusion web UI that we have seen so far—it even has textual inversion and super-resolution models! [github][colab]
@GeeveGeorge shared a notebook that mixes the composition of craiyon with the quality of Stable Diffusion, the results look interesting. [colab][announcement tweet]
@devdef released Stable Warp Fusion 1.5, a notebook to create smooth videos using Stable Diffusion (we share some examples in the AI Art section) [patreon][announcement tweet]
u/pxan created an amazing thread named How to get images that don't suck where the most important concepts for prompting with Stable Diffusion are well explained—such as classifier-free guidance, sampling methods, or seeds [reddit thread]
@johnowhitaker shared Grokking Stable Diffusion, a notebook with simple code to get a better sense of how the different components of Stable Diffusion work—such as the autoencoder, the noise scheduler, or the text encoder—in a hands-on manner. [colab]
Aleksa Gordić created the diffusion playground, a github repository with code to generate, and interpolate images that comes with a video tutorial. [github][video]
🎨 AI Art
normally, people create linear interpolations between the Stable Diffusion encodings of two text prompts (the textual latents), but in these experiments, Justin Pinkney shows his results after interpolating in what he calls the time space.
the way how it works is by combining the information of each text prompt during the sampling of each generation.
in this example, the first image in the interpolation was generated with a prompt like “a painting of a flower” during the whole sampling process, but as the interpolation advances, he progressively changes this prompt at different stages of the sampling with something like “painting of Trump”.
videos created with @devdef ‘s Stable Warp Fusion, by @Infinite__Vibes, @RonnyKhalil, and @itspoidaman.
depth maps experiments, by @CoffeeVectors
creative use of DALL·E to unlock invisibility, by Sergey Galkin.
Xander’s weekend experimentation, exploring the power of random combinations of great prompts.
🦾 krea updates
we have been processing and uploading data to turn krea.ai into the largest search engine of Stable Diffusion generations.
during the process though, some of our users started experiencing issues while saving generations, or annoying delays when searching…
the good news is that during the past three days, we have re-implemented our whole backend, and in the next few hours all these errors should be fixed!
we’ll be announcing new features in our Discord, come say hi!