Executive Summary

Strategists in the United States and China see the potential of artificial intelligence (AI) to enable better, faster decision-making, which will be decisive in future military conflicts. Applications of machine learning will increasingly influence how political and military leaders perceive the strategic environment, weigh risks and options, and judge their adversaries. But what are the risks of exposing critical human decision-making processes to the surprising behaviors and bizarre failures of AI systems?

Reaping the benefits of AI for decision advantage requires first understanding its limitations and pitfalls. AI systems make predictions based on patterns in data. There is always some chance of unexpected behavior or failure. Existing tools and techniques to try and make AI more robust to failures tend to result in trade-offs in performance, solving one problem but potentially worsening another. There is growing awareness of AI vulnerabilities and flaws, but there is also a need for deeper analysis of the potential consequences of technical failures within realistic deployment contexts.

This policy brief examines how failures in AI systems directly or indirectly influencing decision-making could interact with strategic pressures and human factors to trigger escalation in a crisis or conflict:

• Offensive operations incorporating AI or interfering with an adversary’s AI systems could result in unforeseen system failures and cascading effects, triggering accidental escalation.

• AI systems that are insecure, inadequately trained, or applied to the wrong types of problems could inject bad information into decision-making processes, leading to inadvertent escalation.

• Discovery of a compromise of an AI system could generate uncertainties about the reliability or survivability of critical capabilities, driving decision makers toward deliberate escalation if conflict appears imminent.

These scenarios reveal a core dilemma: decision makers want to use AI to reduce uncertainty, especially when it comes to their awareness of the battlefield, knowing their adversaries’ intentions and capabilities, or understanding their own capacity to withstand an attack. But by relying on AI, they introduce a new source of uncertainty in the likelihood and consequences of technical failures in AI systems.

Harnessing AI effectively requires balancing these trade-offs in an intentional and riskinformed approach. There is no way to guarantee that a probabilistic AI system will behave exactly as intended, or that it will give the correct answer. However, militaries can design both AI systems and the decision-making processes reliant upon them to reduce the likelihood and contain the consequences of AI failures, including by:

• Defining a set of mission-specific properties, standards, and requirements for AI systems used in decision-making contexts, such as confidence metrics and safeguards to detect compromise or emergent properties.

• Circumscribing the use of AI in decision-making, applying AI toward narrow questions where it is well suited while reserving for human judgment problems such as interpreting an adversary’s intent; and considering ruling out AI in some areas altogether.

• Involving senior decision-makers as much as possible in development, testing, and evaluation processes for systems they will rely upon, as well as educating them on the strengths and flaws of AI so they can identify system failures.

The United States should continue to lead in setting the global standard for responsible development and use of AI by taking steps to demonstrate certain assurances and, to the extent possible, encourage China to take similar precautions by:

• Clarifying the practices and safeguards implemented to limit the risks of AI systems supporting decision-making.

• Collaborating internationally to develop mutually-beneficial technical safeguards and best practices to reduce the risk of catastrophic AI failures.

• Committing to restraint in the use of offensive operations incorporating AIenabled capabilities and operations targeting AI systems where there are significant escalation risks.