Human monitoring of AI systems will not be as effective as we expect – especially in warfare

As artificial intelligence (AI) becomes more powerful – and is even utilized in warfare – governments, technology corporations and international bodies urgently need to make sure its safety. And a standard thread in most agreements on AI safety is the necessity for human oversight of the technology.

In theory, humans can act as a safeguard against misuse and possible hallucinations (when AI generates false information). For example, this might mean having a human review the content generated by the technology (its output). However, the thought of ​​humans acting as effective controllers of computer systems comes with certain challenges, as demonstrated by a growing body of research and several other examples of AI use within the military.

Many of the efforts so far to create regulations for AI already contain language that advocates human oversight and involvement. For example: the EU AI law requires that high-risk AI systems – for instance those already in use that mechanically discover people using biometric technologies comparable to a retinal scanner – should be individually reviewed and confirmed by no less than two individuals with the obligatory competence, training and authority.

In the military field, the importance of human oversight was highlighted by the UK government in its February 2024 response to a parliamentary report on AI in weapon systemsThe report emphasizes “meaningful human control” by providing individuals with appropriate training. It also emphasizes the thought of ​​human responsibility and says that decision-making in actions comparable to armed drones can’t be delegated to machines.

This principle has largely been maintained up to now. Military drones are currently controlled by human operators and their chain of command, who’re answerable for the actions of an armed aircraft. However, artificial intelligence has the potential to make drones and the pc systems they use more powerful and autonomous.

This includes their targeting systems. In these systems, AI-controlled software selects and locks on enemy combatants in order that humans can authorize a weapon attack against them.

Although this technology just isn’t yet widely used, the war in Gaza seems to have shown that it’s already getting used. The Israeli-Palestinian publication +972 magazines reported described a system called Lavender utilized in Israel, which is reportedly an AI-based destination advice system that’s coupled with other automated systems and tracks the geographic location of identified targets.

Target acquisition

In 2017, the US military designed a project called Mavenwith the aim of integrating AI into weapon systems. Over the years, it has evolved right into a goal acquisition system. It has reportedly significantly increased the efficiency of the goal advice process for weapon platforms.

In line with recommendations from scientific work on AI ethics, a human is present to watch the outcomes of the targeting mechanisms as a critical a part of the decision-making process.

Nevertheless, work on the psychology of human collaboration with computers raises necessary questions that have to be considered. In a 2006 peer-reviewed articleUS scientist Mary Cummings summarised how people can come to put excessive trust in machine systems and their conclusions – a phenomenon often known as Automation bias.

This has the potential to undermine the human role in overseeing automated decision-making when operators are less more likely to query a machine's conclusions.

In one other study In a study published in 1992, researchers Batya Friedman and Peter Kahn argued that the sense of ethical agency when interacting with computer systems may be weakened to the purpose where people don’t feel answerable for the resulting consequences. In fact, the paper explains that folks may even begin to attribute a way of agency to the pc systems themselves.

Given these trends, it might be sensible to think about whether excessive reliance on computer systems, and the potential weakening of human moral self-confidence that this entails, could even have implications for targeting systems. For although the error rate is statistically small on paper, it takes on frightening proportions after we consider the potential impact on human lives.

The various resolutions, agreements and laws on AI help to supply certainty that humans will act as a very important control authority for AI. However, it will be significant to ask whether a human will find a way to proceed to function a very important controller of AI after a protracted period on this role. The connection could also be interrupted whereby human operators begin to perceive real people as objects on a screen.