Categories
Uncategorized

The Future of AI Images

How a model of the world can make a more believable breaking wine glass

Last year I thought that AI images had more or less hit a plateau in terms of what they could offer. Like the quality was, in the best of them, as photo-realistic (or paint-realistic, illustration-realistic, etc) as they could be. There were some problems still to be solved, such as control over composition and pose, but I didn’t think there were more breakthroughs to be had. I thought we were basically done.

I was proved wrong by Google’s Nano Banana Pro. Its breakthrough was to integrate image generation with the reasoning of Gemini 3, so you can create much more exactly what you want. You can edit with your words, it preserves image states much better, you can use Gemini to ask it to “fill in the blanks” of your prompts… basically it reads your intent and ‘understands’ you better. It creates much more of a close relationship between you and the model; you don’t have to know what to ask for in its language, you can use your language and it gets it really well and is able to do very complex outputs including text and infographics.

That was a breakthrough I hadn’t foreseen, and it made me wonder what else I was missing. What’s next for image models?

The Falling Wine Glass Problem

I think it comes down to this: if you ask an image model, no matter how advanced, to generate a picture of a wine glass full of red wine hitting the floor and spilling wine it will give you all of the ingredients visually, but assembled incorrectly. The physics will be wrong; the wine will already be on the carpet or will be spilling out of the glass in an unrealistic way.

Six images of a glass of red wine falling on a white rug; all of them are wrong in some way
Prompt: A glass of red wine falling onto a white shag rug, captured the millisecond before impact.

Image models create visual mimicry; they know what elements should be in the scene, but none of the actions happen in the correct sequence (except perhaps by luck). And so I think that’s the next step: causal understanding. Cause and effect, action and reaction.

Changes Over Time

Maybe image models need to ‘imagine’ before they generate. NVIDIA recently released an experimental image model called ChronoEdit which uses temporal reasoning; when given an image and prompted for a change it will (kind of) imagine running the changes forward through time, like a video, to reach the end state and return the final frame as the image result. For example, given an image of an empty room and asked to “insert a metal object ball with smooth curved surfaces into the scene” it imagines the ball rolling into place and then shows you its final position.

This is a quite slow and intensive approach, but its advantage is a result which should be much more consistent with physics as we know and observe it.

But this ‘imagined video’ approach isn’t a perfect solution because video models themselves struggle with causality. As with images they’ll get some things right to a certain degree but they’ll also get things very wrong — like the falling glass.

Although advanced models such as Veo 3 have already exhibited some learning of physics — for example, the difference in reaction when dropping something light or something heavy in water — they often get confused in complex scenes where there’s more than one thing happening. They might ‘know’ the gravity of a glass falling, and a bit about how fluid should move, but they still struggle with the sequence: the glass falls, it hits the ground, the liquid explodes out. They don’t understand that one thing causes the next.

Enter World Models

You may have noticed that recently I’ve been posting more about world models — that is, AI models that are trained (largely on video) to the extent that they begin to ‘understand’ the physical laws of an environment, such as a video game or — ideally — our reality.

Some world models only generate a place, a persistent 3D space that can be moved around in, but not interacted with. But with highly complex and powerful training, they can generate an experience, a persistent space that can also be interacted with; they can learn not just what motion looks like, but how it works. Rather than 2D mimicking 3D, they create a latent space in 4D, with some understanding of motion over time. Probably the most advanced in that area at the moment is Google’s Genie 3:

The experiences it generates look quite simple — a character moving around in a world space — but have some understanding of physics; for example, if you make a character walk towards a wall it will stop and be unable to pass through it because that breaches the laws of physics. As the technology progresses, future versions will hopefully learn more physical laws, and cause and effect too.

World models have lots of potential uses, from prototyping video game levels to training cars and robots. And they’ll clearly be useful for generating videos, but will also be very good for generating images.

From Visual Mimicry to Causal Understanding

Returning to my earlier example, if I prompt for a picture of a glass of red wine falling on the carpet and spilling then a world model should be able to “imagine” it using its understanding of causality and then generate an image that’s essentially a frozen moment in time, depicted in a way that makes sense to us. That might be slow and intensive, but these things tend to become faster and more efficient over time.

So that, I think, is the next potential breakthrough for image models: a baked-in understanding of how the world works, its physics and causality. Rather than being just a visual representation of a scene with all the ingredients but in the wrong place in the wrong order, everything will just look right to us.

I don’t know when this is going to happen, but I imagine we’ll start to see it more this year.


Post-publishing updates

PhysicEdit is a new experimental approach to physics in image models, using a version of QwenImage Edit fine-tuned with 38,000 videos of physical state transitions.

Nodes Aren’t the Future of AI Creation. Here’s What Is. Bilawal Sidhu on the shortcomings of current creative tools, and the approaches that we’ll likely adopt in the future.

Categories
Uncategorized

AI as Normal Technology

I’m reading the report AI as Normal Technology. I haven’t finished it yet (it’s long!) but its thesis is that, despite all the noise and hype, AI is progressing on the path of every technology before it, even the highly transformative ones such as electricity and the internet, and doesn’t need to be thought of as a special case.

This is the most important chart from it:

There has been (and still is) a lot of invention, a lot of innovation, and a lot of adoption; 58% of employees intentionally use AI at work, according to this survey. But we’re still in the early stages of diffusion: it hasn’t become so integral to the ways we work that we’ve adapted our workflows to make the most of it.

Categories
Uncategorized

Is Firefly 2 a first-class image generation model?

Adobe just released the second version of Firefly, its image generation model. While models such as Stable Diffusion and Midjourney are the market leaders in terms of result quality, the uncertainty (warranted or otherwise) about the datasets they’ve used in their training means that it’s important to have strong rivals that are trained on licensed data.

I had high hopes for Firefly when it was released, but unfortunately it fell short in many areas. So is Firefly 2 any better? In short, yes. But is it good enough to be a viable contender? Well, let’s take a look.

Categories
Uncategorized

Be weird to diversify Midjourney images

We know that the data used to train foundational AI models perpetuates bias, and in Midjourney that most clearly manifests1 in results involving people; unless explicitly instructed otherwise, the people in generated images tend to look beautiful.

To show you what I mean here’s a prompt I wrote, with the resulting generated images below it:

candid photograph, close-up, a young Korean woman with bleached blonde hair, bright natural light, happy, joyful --style raw

The results look great, but clearly fall squarely within conventional standards of beauty. Not everybody looks this beautiful (myself included), and we should reflect that.