gen‑ai.news
← Back
Multimodal

New benchmark confirms AI models still perform poorly at visual perception

New benchmark confirms AI models still perform poorly at visual perception

Moonshot AI has released PerceptionBench, a benchmark designed to measure how well multimodal AI models can genuinely interpret visual information - separate from their capacity for logical reasoning. The distinction matters because most existing evaluations bundle the two abilities together, making it difficult to identify where a model actually fails. PerceptionBench attempts to isolate the perception step, testing whether a model can correctly extract what is in an image before any higher-order thinking takes place.

The results across frontier models are consistently weak. No model in the evaluation reaches 60 percent accuracy on the benchmark, and the top performer, GPT-4o Sol, holds only a narrow lead over its competitors. That kind of ceiling, across models from multiple major labs, points to a shared and fundamental limitation rather than a gap that any one team has quietly solved.

One of the more significant findings is that errors researchers have long labeled as reasoning failures may be misattributed. If a model misreads key details in an image at the perception stage, any downstream reasoning built on that flawed input will also be wrong - but the mistake looks, from the outside, like a logic error. PerceptionBench's structure makes it possible to catch failures earlier in that chain, which could shift how researchers prioritize improvements to multimodal architectures.

The benchmark adds to a growing body of work questioning how deeply current vision-language models actually "see" versus pattern-match on statistical regularities in training data. For developers building applications that depend on accurate visual understanding - medical imaging tools, document analysis, autonomous systems - the results are a useful reminder that model confidence and model accuracy are not the same thing. Closing the perception gap may require architectural changes or new training approaches that go beyond simply scaling existing systems.

Enjoy this story? Get the next one in your inbox.

Twice a week: the most important stories in generative image and video AI, distilled into a 2-minute read.

Free. Unsubscribe any time. No spam, ever.

Your next read

What Is MAX Sneaks? Your First Look at Tools That Don’t Exist Yet
Multimodal

What Is MAX Sneaks? Your First Look at Tools That Don’t Exist Yet

Adobe MAX Sneaks is an annual event where Adobe researchers demo experimental technology that has not yet made it into any product. Some of these demos eventually become familiar tools in Premiere, After Effects, or Photoshop - others never ship at all. This year's Sneaks airs Wednesday, November 11, and is free to watch online.

Adobe’s AI experiments let you walk around in photos and interview your characters
Multimodal

Adobe’s AI experiments let you walk around in photos and interview your characters

Adobe's annual Max conference has brought a fresh batch of experimental features, including tools that convert flat photos into navigable 3D scenes, let creators shape AI characters through conversation, and simplify photo editing to a copy-and-paste workflow. These "Sneaks" projects offer a glimpse into where Adobe's generative AI research is heading. None are shipping products yet, but they reflect the direction of the company's broader creative toolset.

Tom Hanks Played Five Characters in This 2004 Christmas Movie
Multimodal

Tom Hanks Played Five Characters in This 2004 Christmas Movie

The Polar Express, Robert Zemeckis's 2004 performance-capture film, is back in seasonal rotation - and it remains one of the more technically unusual productions of its era. Tom Hanks played five distinct characters across 38 days of work, a casting choice that was rooted in story logic as much as technical curiosity. Here is a closer look at how that decision came together and what it meant for the actors involved.