A false intelligence report created with the help of an artificial intelligence chatbot nearly led the US military to intercept a Chinese vessel during the US-Iran conflict earlier this year, highlighting the risks of using generative AI in high-stakes intelligence work.
According to reports citing a CNN investigation, an analyst working with a US special operations command used an AI chatbot while assessing intelligence related to the cargo manifest of a Chinese ship travelling through the Middle East.
The chatbot reportedly combined information from open sources with classified signals intelligence but incorrectly identified the material being transported. The resulting assessment claimed the vessel was carrying components linked to a nuclear weapons programme.
The information was subsequently incorporated into an intelligence report that circulated within the US military. The assessment was treated seriously enough for preparations to begin for an interception of the Chinese vessel.
Military aircraft were reportedly already in the air and armed personnel were preparing to board the ship when officials conducted a closer review of the underlying intelligence. The review found that the central claim about the vessel's cargo was incorrect, and the planned operation was called off before an interception took place.
The episode was described by sources familiar with the incident, while details about the specific AI system involved have not been publicly disclosed. The Pentagon has also not provided a detailed public account of the reported incident.
The case draws attention to AI hallucination, a known limitation of generative AI systems in which models can produce information that appears convincing but is inaccurate or unsupported. In intelligence environments, such errors can carry greater consequences when AI-generated information feeds into operational decisions.
The incident does not indicate that an AI system independently decided to intercept the vessel. Instead, it shows how incorrect information generated with AI assistance can enter a human decision-making process if the underlying claims are not independently verified.
Generative AI is increasingly being explored by defence and intelligence organisations for analysing large volumes of information, identifying patterns and accelerating research. However, the reported incident illustrates the importance of maintaining human verification when AI-generated outputs are used in sensitive environments.
The interception was ultimately stopped before military action was taken. However, the episode adds to the debate around safeguards, verification processes and accountability as governments integrate generative AI into national security and intelligence operations.
Disclaimer: This article may include information derived from interviews, press releases, public statements, research, company communications and other publicly available or third-party sources. Such material may be summarised, paraphrased or contextualised for journalistic and editorial purposes. All rights in third-party content remain with their respective owners.