How AI Creates Photorealistic Images From Text

Have you ever imagined a baby dog hatching from a broken egg? Or a picture showing a city filled with steam-powered airships? How about two robots having a romantic date at the movies? These might seem like wild ideas, but there’s a special kind of computer technology called “text-to-image generation” that can make them real. These computer programs can create pictures that look like real photos from simple written descriptions.

At Google Research, our scientists and engineers have been exploring how to make text-to-image technology better. We’ve recently introduced two new models for this, called Imagen and Parti. Both of them can create very realistic pictures, but they do it in different ways. Let’s take a closer look at how these models work and what they can do.

How Text-to-Image Models Work:

Text-to-image models are like magic artists. You tell them what you want to see, and they try to draw it for you. You can give them easy instructions like “draw an apple” or more complicated ones like “draw a cute sloth holding a small treasure chest with a bright golden glow coming from it.” These models have gotten better because they’ve learned from lots of pictures and their descriptions. This has led to big advances in this field, like OpenAI’s DALL-E

How Imagen and Parti Work:

Imagen and Parti are like different kinds of art tools. They both use something called Transformer models to understand how words in a sentence relate to each other. But they also use unique techniques to make pictures that match the written descriptions.

Imagen is like a magic painting machine. It starts with blurry pictures and makes them clearer step by step. This is called a Diffusion model. It’s great at things like improving picture quality, changing black-and-white photos to color, and other cool tricks.

Parti is more like a puzzle solver. It turns a bunch of pictures into a code, like puzzle pieces. When you give it a written description, it uses this code to create a new picture. This helps when you have long and complex descriptions

These Models Have Some Limits:

As cool as they are, Imagen and Parti can’t do everything perfectly. They struggle with tasks like counting objects accurately or arranging them just right based on words like “to the left of” or “on top of.” When the descriptions get very complicated, they might miss some details or add things that weren’t even mentioned. This happens because they have some limitations, like not knowing about 3D space. We’re working on making them better at these things.

Being Responsible with Imagen and Parti:

These text-to-image models are amazing tools, but they also come with some concerns. They could be used to spread false information, show biases, or cause problems. So, we’re taking responsible steps. We put special marks on images made by Imagen and Parti so people can recognize them. We’re also studying how these models might have biases, especially in how they represent people and cultures, and trying to fix them. You can read more about these issues in the Imagen and Parti papers.

What’s Coming Next:

We’re not stopping here. We want to make these models even better by combining the best parts of Imagen and Parti. We’re also thinking about letting people create and edit pictures using text, like a creative conversation with a computer. And, we’ll keep checking how well these models work to make sure they follow our principles of being responsible and safe. Our goal is to share these models with the world in a way that sparks creativity and keeps everyone safe.