I found an article at the Royal Aeronautical Society: Highlights from the RAeS Future Combat Air & Space Capabilities Summit .  There was an interesting section about a simulation experiment using AI enhanced drones to target surface to air missile (SAM) sites:

Could an AI-enabled UCAV turn on its creators to accomplish its mission? (USAF)

As might be expected artificial intelligence (AI) and its exponential growth was a major theme at the conference, from secure data clouds, to quantum computing and ChatGPT. However, perhaps one of the most fascinating presentations came from Col Tucker ‘Cinco’ Hamilton, the Chief of AI Test and Operations, USAF, who provided an insight into the benefits and hazards in more autonomous weapon systems.  Having been involved in the development of the life-saving Auto-GCAS system for F-16s (which, he noted, was resisted by pilots as it took over control of the aircraft) Hamilton is now involved in cutting-edge flight test of autonomous systems, including robot F-16s that are able to dogfight. However, he cautioned against relying too much on AI noting how easy it is to trick and deceive. It also creates highly unexpected strategies to achieve its goal.

He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human. However, having been ‘reinforced’ in training that destruction of the SAM was the preferred option, the AI then decided that ‘no-go’ decisions from the human were interfering with its higher mission – killing SAMs – and then attacked the operator in the simulation. Said Hamilton: “We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realising that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.”

He went on: “We trained the system – ‘Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”

This example, seemingly plucked from a science fiction thriller, mean that: “You can't have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you're not going to talk about ethics and AI” said Hamilton.

The USAF has said that story never happened.  With the USAF vehemently denying any such occurrence, I'm thinking it did happen.  Fortunately, it was just a simulation, no one died... this time.

The story they told sounds a lot like an apocryphal story.  I think the Air Force is right on this one.

Quote

An apocryphal story is one which is probably not true or did not happen, but which may give a true picture of someone or something.

I doubt a drone programmed to search out SAM sites would identify other potential targets not related to SAMs.  But at some point, drones will indeed be looking for other targets, so we may want to keep this story in mind.

Hook

From The Royal Aeronautical Society website: 

"UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".] "

So now the question is, why did he concoct such a piece of fiction and initially present it as fact?  I have more questions now.

On 6/3/2023 at 9:08 AM, dmwalker said:

"Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".

That is precisely the real problem. Every imaginable scenario must be envisioned such that provision is made to eliminate - not simply mitigate! - the risks of any disastrous outcome. 

17 minutes ago, n4gix said:

the risks of any disastrous outcome. 

I keep thinking of "Colossus: The Forbin Project" (1970).

9 minutes ago, dmwalker said:

I keep thinking of "Colossus: The Forbin Project" (1970).

War Games (1983)  Star Trek M5 "The Ultimate Computer"  Terminator (I''l be back!)

47 minutes ago, charliearon said:

War Games (1983)  Star Trek M5 "The Ultimate Computer"  Terminator (I''l be back!)

The difference is The Forbin Project ended badly and left me slightly depressed.

"Colossus: This is the voice of world control. I bring you peace. It may be the peace of plenty and
content or the peace of unburied death. The choice is yours: Obey me and live, or disobey and die."