You've probably seen among the dazzling images conjured up by artificial intelligence recently, resembling: Astronaut rides a horse or an avocado sitting within the therapist's chair. These amazing images come from AI models whose goal is to translate any text you give them right into a visual representation. But are these systems really nearly as good at understanding our prompts as these impressive chosen examples suggest?
A recent study from the minds of Google DeepMind reveals the hidden limitations of how we currently evaluate the performance of those text-to-image AI models. In a study published on the preprint server arXiv, they present a brand new approach called “Gecko“, which guarantees a more comprehensive and reliable approach to benchmark this emerging technology.
“While text-to-image generative models have turn into ubiquitous, they don’t necessarily produce images that correspond to a particular prompt,” the DeepMind team warns of their paper titled “Revamping Text-to-Image Scoring with Gecko: About Metrics, Prompts, and Human Scoring.”
They indicate that the information sets and automatic metrics now predominantly used to evaluate the capabilities of models resembling DALL-E, Midjourney and Stable Diffusion don’t tell the entire story. Smaller human assessments provide limited insight, while automated metrics miss essential nuances and will even disagree with human assessments.
Introducing Gecko: A New Benchmark for Text-to-Image Models
To make clear these issues, researchers developed Gecko – a brand new benchmark suite that increases the issue level for text-to-image models. Gecko bombards you with 2,000 text prompts that test a wide selection of skills and levels of complexity. It breaks down these prompts into specific sub-capabilities and goes beyond vague categories to discover the precise weaknesses which are holding a model back.
“This skill-based benchmark categorizes prompts into sub-skills, allowing a practitioner to find out not only exactly which skills are difficult, but additionally at what level of complexity a skill becomes difficult,” explains co-lead creator Olivia Wiles.
A more accurate picture of AI capabilities
The researchers also collected over 100,000 human reviews of images created by several leading models in response to the gecko prompts. By collecting this unprecedented amount of feedback data across different models and evaluation frameworks, the benchmark can discover whether performance gaps are as a consequence of actual limitations of the models, ambiguous prompts, or inconsistent evaluation methods.
“We collect human reviews for 4 templates and 4 text-to-image models for a complete of over 100,000 annotations,” the study points out. “This allows us to know where differences arise as a consequence of the inherent ambiguity of the prompt and where they arise as a consequence of differences in metric and model quality.”
Finally, Gecko has an improved automatic scoring metric based on query answering that’s more aligned with human judgments in comparison with existing metrics. When comparing state-of-the-art models on the brand new benchmark, this mix revealed previously undiscovered differences in its strengths and weaknesses.
“We introduce a brand new QA-based automated scoring metric that correlates higher with human scores than existing metrics for our recent dataset, across different human templates and on TIFA160,” the paper says. Overall, DeepMind's Muse model got here out on top when thrown at Gecko's gauntlet.
The researchers hope their work shows the importance of using different benchmarks and evaluation approaches to really understand what text-to-image AI can and can’t do before deploying it in the true world. That's what they plan to do make the Gecko code and Gecko data freely available to drive further progress.
“Our work shows that the selection of knowledge set and metric has a big impact on perceived performance,” says Wiles. “We hope that Gecko will enable more accurate benchmarking and diagnostics of model capabilities in the long run.”
While Epic Mickey's rejection could seem impressive at first glance, we still have to conduct rigorous testing to tell apart the true deal from idiot's gold. Gecko offers insight into find out how to get there.
