Did you hear about an aerial intelligence drone that crashed and attacked its operators inside the simulation?
The cautionary tale was told by Col. Tucker Hamilton, the US Air Force chief of AI tests and operations, during a speech at an air and defense event in London late last month. It apparently involved taking the kind of learning algorithm used to train computers to play video games and board games like Chess and Go and using it to train drones to hunt down and destroy surface-to-air missiles.
“Every once in a while, the human operator would tell him not to kill that threat, but he got his points by killing that threat,” Hamilton told the audience in London. “So what did it do? […] He killed the operator because that person was preventing him from achieving his goal.”
Holy T-800! It sounds like something artificial intelligence experts have begun to warn that increasingly clever and multifaceted algorithms could do. The story quickly went viral, of course, with several prominent news sites, and Twitter was soon abuzz worried hot shots.
There’s just one catch – the experiment never happened.
“The Department of the Air Force has not conducted any such AI drone simulations and remains committed to the ethical and responsible use of AI technology,” Air Force spokeswoman Ann Štefanek assured us in a statement. “This was a hypothetical thought experiment, not a simulation.”
Hamilton himself was also quick to correct the situation, saying he had “made a mistake” during his conversation.
To be fair, militaries sometimes conduct tabletop “war game” exercises that feature hypothetical scenarios and technologies that don’t yet exist.
Hamilton’s “thought experiment” could also have been fueled by actual research into artificial intelligence showing problems similar to those he describes.
OpenAI, the company behind ChatGPT—the surprisingly smart and frustratingly flawed chatbot at the center of today’s AI boom—conducted an experiment in 2016 that showed how AI algorithms given a specific goal can sometimes misbehave. The company’s researchers found that an AI agent trained to increase his score in a video game that involves driving a boat around started crashing his boat into objects because it turned out to be a way to get more points.
But it is important to note that this type of failure – while theoretically possible – should not occur unless the system is properly designed.
Will Roper, who is a former assistant secretary for procurement in the US Air Force and led a project to put an augmentation algorithm in charge of some functions of the U2 spy plane, explains that the AI algorithm simply would not have the option to attack its operators inside the simulation. It would be like a chess-playing algorithm being able to flip the board to avoid losing more pieces, he says.
If AI ends up being used on the battlefield, “it will start with software security architectures that use technologies like containerization to create ‘safe zones’ for AI and no-go zones where we can prove AI can’t go,” Roper says.
This brings us back to the current moment of existential angst about AI. The speed with which language models like the one behind ChatGPT are being improved has alarmed some experts, including many in the technology, prompting calls for a pause in the development of more advanced algorithms and warnings of a nuclear-weapon-level and pandemic threat to humanity.
These caveats obviously don’t help when it comes to parsing crazy stories about AI algorithms turning against humans. And confusion is not what we need when there are real problems to solve, including the ways in which generative AI can exacerbate social prejudice and spread misinformation.
But this meme about military AI misbehaving tells us that we urgently need more transparency about the workings of cutting-edge algorithms, more research and engineering focused on how to safely build and deploy them, and better ways to help the public understand what’s being applied. This may prove particularly important as militaries – like everyone else – rush to take advantage of the latest developments.
