In 1994, Florida jewelry designer Diana Duyser discovered the image of the Virgin Mary in a grilled cheese sandwich, which she preserved and later auctioned off for $28,000. But how much do we actually understand about pareidolia, the phenomenon of seeing faces and patterns in objects regardless that they aren't really there?
A brand new one study from the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) addresses this phenomenon and presents a big, human-labeled dataset of 5,000 pareidolic images that far exceeds previous collections. Using this data set, the team discovered several surprising results in regards to the differences between human and machine perception and the way the flexibility to see faces in a slice of toast can have saved the lives of distant relatives.
“Facial pareidolia has long fascinated psychologists, however it stays largely unexplored in the pc vision community,” says Mark Hamilton, an MIT doctoral student in electrical engineering and computer science, a CSAIL partner and lead researcher on the work. “We desired to create a resource that might help us understand how each humans and AI systems process these illusory faces.”
What did all those fake faces reveal? For one thing, AI models don't seem to acknowledge pareidolic faces the way in which we do. Surprisingly, the team found that only by training algorithms to acknowledge animal faces did they grow to be significantly higher at recognizing pareidolic faces. This unexpected connection suggests a possible evolutionary link between our survival-critical ability to acknowledge animal faces and our tendency to see faces in inanimate objects. “A result like this seems to suggest that pareidolia may arise not from human social behavior, but from something deeper: corresponding to quickly spotting a lurking tiger or recognizing which way a deer is facing, so our ancient ancestors could hunt,” says Hamilton.
Another fascinating discovery is what the researchers call the “Goldilocks zone of pareidolia,” a category of images during which pareidolia is more than likely to occur. “There is a specific range of visual complexity during which each humans and machines are more than likely to perceive faces in non-face objects,” says William T. Freeman, MIT professor of electrical engineering and computer science and principal investigator on the project. “Too easy and there isn't enough detail to form a face. Too complex and it becomes visual noise.”
To uncover this, the team developed an equation that models how humans and algorithms recognize illusory faces. Analyzing this equation, they found a transparent “pareidolic peak” where the likelihood of seeing faces is highest, corresponding to pictures which have “just the best amount” of complexity. This predicted “Goldilocks Zone” was then validated in tests with each real human subjects and AI facial recognition systems.
This recent record: “Faces in things“dwarfs the outcomes of previous studies, which generally only used 20-30 stimuli. This scale allowed researchers to look at how state-of-the-art facial recognition algorithms behaved after fine-tuning on pareidolic faces, and showed that these algorithms couldn’t only be edited to acknowledge these faces, but that they might also act as a silicon proxy for ours own brain, allowing the team to ask and answer questions on the origins of pareidolic facial recognition which are unimaginable to ask in humans.
To create this dataset, the team curated roughly 20,000 candidate images from the LAION-5B dataset, which were then rigorously labeled and assessed by human annotators. This process involved drawing bounding boxes around perceived faces and answering detailed questions on each face, corresponding to the perceived emotion, age, and whether the face was accidental or intentional. “Collecting and annotating 1000’s of images was a frightening task,” says Hamilton. “Much of the dataset owes its creation to my mother,” a retired banker, “who spent countless hours lovingly labeling images for our evaluation.”
The study also has potential applications in improving facial recognition systems by reducing false positives, which could have implications for areas corresponding to self-driving cars, human-computer interaction and robotics. The data set and models could also help areas corresponding to product design, where understanding and controlling pareidolia could produce higher products. “Imagine having the ability to routinely optimize the design of a automotive or a toddler’s toy to make it look friendlier, or make sure that a medical device doesn’t unintentionally appear threatening,” says Hamilton.
“It is fascinating how people instinctively interpret inanimate objects as having human-like features. For example, should you take a have a look at an electrical outlet, you’ll be able to immediately imagine it singing, and you’ll be able to even imagine it “moving its lips.” However, algorithms don’t inherently recognize these cartoon faces in the identical way we do,” says Hamilton. “This raises interesting questions: What explains this difference between human perception and algorithmic interpretation? Is pareidolia useful or harmful? Why don’t algorithms experience this effect like we do? These questions prompted our investigation, as this classic psychological phenomenon in humans has not yet been thoroughly investigated in algorithms.”
As researchers prepare to share their data set with the scientific community, they’re already looking ahead. Future work could include training visual language models to grasp and describe pareidic faces, potentially resulting in AI systems that may cope with visual stimuli in a more human-like manner.
“This is a stunning essay! It's fun to read and makes me think. Hamilton et al. Ask an intriguing query: “Why can we see faces in things?” says Pietro Perona, the Allen E. Puckett Professor of Electrical Engineering at Caltech, who was not involved within the work. “They emphasize that learning from examples, including animal faces, only goes half the strategy to explaining the phenomenon.” I bet that desirous about this query will teach us something necessary about how our visual system evolves beyond training it maintains over the course of life, generalizes.”
Hamilton and Freeman's co-authors include Simon Stent, a research scientist on the Toyota Research Institute; Ruth Rosenholtz, senior research scientist within the Division of Brain and Cognitive Sciences, NVIDIA research scientist and former CSAIL member; and CSAIL partner postdoc Vasha DuTell, Anne Harrington MEng '23 and research scientist Jennifer Corbett. Her work was supported partially by the National Science Foundation and the CSAIL MEnTorEd Opportunities in Research (METEOR) Fellowship, while she was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator. MIT SuperCloud and the Lincoln Laboratory Supercomputing Center provided HPC resources for the researchers' results.
This work shall be presented this week on the European Conference on Computer Vision.
