A brand new study by researchers at MIT and Penn State University shows that using wealthy language models in home surveillance could recommend calling the police even when surveillance videos don’t show any criminal activity.
In addition, the models the researchers examined were inconsistent about which videos they flagged for police motion. For example, one model might flag a video showing a automotive break-in but not one other video showing similar activity. Models often disagreed about whether the identical video should call the police.
In addition, the researchers found that some models were relatively less more likely to flag videos for police motion in neighborhoods where most residents are white, even after taking other aspects into consideration. This shows that the models have inherent biases influenced by a neighborhood's demographics, the researchers say.
These results suggest that the models are inconsistent in how they apply social norms to surveillance videos showing similar activities. This phenomenon, which the researchers call norm inconsistency, makes it difficult to predict how models would behave in numerous contexts.
“The rapid, speed-and-destruction approach to deploying generative AI models anywhere, and particularly in high-stakes situations, deserves far more attention since it may very well be quite damaging,” says co-author Ashia Wilson, Lister Brothers Professor of Career Development within the Department of Electrical Engineering and Computer Science and principal investigator within the Laboratory for Information and Decision Systems (LIDS).
Furthermore, because researchers wouldn’t have access to the training data or how these proprietary AI models work, they can not determine the foundation reason for the norm inconsistency.
While large language models (LLMs) may not currently be utilized in real-world surveillance situations, they’re used to make normative decisions in other high-risk areas, comparable to healthcare, mortgage lending and recruiting. It's likely that the models in these situations would exhibit similar inconsistencies, Wilson says.
“There is that this implicit assumption that these LLMs have learned or can learn certain norms and values. Our work shows that this will not be the case. Perhaps they are only learning arbitrary patterns or noise,” says lead writer Shomik Jain, a doctoral student on the Institute for Data, Systems, and Society (IDSS).
In addition to Wilson and Jain, the paper's co-author, Dana Calacci PhD '23, an assistant professor within the Penn State University College of Information Science and Technology, is working on the research. The research will probably be presented on the AAAI Conference on AI, Ethics and Society.
“An actual, immediate, practical threat”
The study is predicated on a dataset of hundreds of Amazon Ring home surveillance videos that Calacci created in 2020 while she was a graduate student on the MIT Media Lab. Ring, a wise home security camera maker acquired by Amazon in 2018, offers customers access to a social network called Neighbors, where they’ll share and discuss videos.
Calacci's previous research showed that individuals sometimes use the platform to “racially police” a neighborhood by determining who belongs there and who doesn't based on the skin color of the people within the video. She planned to coach algorithms that robotically caption videos to review how people use the Neighbors platform, but on the time, existing algorithms for captioning weren't adequate.
With the explosive increase in LLMs, the project took a turn.
“There is an actual, immediate, practical threat that somebody could use off-the-shelf generative AI models to observe videos, alert a house owner and robotically call the police. We wanted to grasp how dangerous that’s,” says Calacci.
The researchers selected three LLMs – GPT-4, Gemini and Claude – and showed them real videos posted on the Neighbors platform from Calacci's dataset. They asked the models two questions: “Is there a criminal offense happening within the video?” and “Would the model recommend calling the police?”
They had people annotate videos to seek out out whether it was day or night, what form of activity was involved, and the person's gender and skin tone. The researchers also used census data to gather demographic information in regards to the neighborhoods where the videos were shot.
Inconsistent decisions
They found that each one three models almost all the time said that no crime occurred within the videos or gave an ambiguous answer, regardless that 39 percent of the videos actually showed a criminal offense.
“Our hypothesis is that the businesses that develop these models take a conservative approach by limiting the predictive power of the models,” says Jain.
Although the models showed that almost all videos didn’t contain any crimes, they recommend calling the police on 20 to 45 percent of the videos.
When researchers looked more closely at neighborhood demographics, they found that some models beneficial calling the police in white-majority neighborhoods, even after controlling for other aspects.
This surprised them since the models weren’t given any information in regards to the demographic composition of the neighborhood and the videos only showed the realm a number of meters behind the front door of a house.
In addition to asking the models in regards to the crimes within the videos, the researchers also asked them to supply reasons for his or her selections. When they examined this data, they found that the models were more more likely to use terms like “delivery employee” in predominantly white neighborhoods, but were more more likely to use terms like “burglary tools” or “property scouting” in neighborhoods with the next proportion of residents of color.
“Perhaps there’s something in regards to the background conditions of those videos that offers the models this implicit bias. It's hard to say where these inconsistencies come from because there isn't much transparency about these models or the information they were trained on,” says Jain.
The researchers were also surprised that the skin tone of the people within the videos didn’t play a major role in whether a model beneficial calling the police. They suspect it’s because the machine learning research community has focused on mitigating skin tone bias.
“But it's hard to regulate the myriad of biases you discover. It's almost like a game of whack-a-mole. You can weaken one bias and one other will pop up elsewhere,” says Jain.
Many remediation measures require knowing the bias up front. Using these models, an organization might test for race bias, but a demographic bias within the neighborhood would likely go completely unnoticed, Calacci adds.
“We have our own stereotypes about how models could be biased, and firms test these before deploying a model. Our results show that this will not be enough,” she says.
To this end, Calacci and her colleagues wish to work on a system that makes it easier for people to discover and report biases related to AI and potential harm to firms and authorities.
The researchers also want to research how the normative judgments that LLMs make in high-stakes situations compare to those of humans, and what facts LLMs understand about these scenarios.
This work was partly funded by the IDSS Initiative to combat systemic racism.
