As artificial intelligence begins to transform cybersecurity, the pressure to adapt may put competing states on a collision course. Recent advances in machine learning techniques could enable groundbreaking capabilities in the future, including defenses that automatically interdict attackers and reshape networks to mitigate offensive operations. Yet even the most robust machine learning cyber defenses could have potentially fatal flaws that attackers can exploit. Rather than end the cat-and-mouse game between cyber attackers and defenders, machine learning may usher in a dangerous new chapter.

Could embracing machine learning systems for cyber defense actually exacerbate the challenges and risks of cyber competition? This study aims to demonstrate the possibility that machine learning could shape cyber operations in ways that drive more aggressive and destabilizing engagements between states. While this forecast is necessarily speculative, its purpose is practical: to anticipate how adversaries might adapt their tactics and strategies, and to determine what challenges might emerge for defenders. It derives from existing research demonstrating the challenges machine learning faces in dynamic environments with adaptive adversaries.

This study envisions a possible future in which cyber engagements among top-tier actors come to revolve around efforts to target attack vectors unique to machine learning systems or, conversely, defend against attempts to do so. These attack vectors stem from flaws in machine learning systems that can render them susceptible to deception and manipulation. These flaws emerge because of how machine learning systems “think,” and unlike traditional software vulnerabilities, they cannot simply be patched. This dynamic leads to two propositions for how these attack vectors could shape cyber operations.

The first proposition concerns offense: Attackers may need to intrude deep into target networks well in advance of an attack in order to circumvent or defeat machine learning defenses. Crafting an attack that can reliably deceive a machine learning system requires knowing a specific flaw in how the system thinks. But discovering such a flaw may be difficult if the system is not widely exposed or publicly available. To reach a hardened target, an attacker may try to compromise the system during development. An attacker with sufficient access could reverse-engineer a system during its development to discover a flaw or even create one by sabotaging the process. This opportunity to gain intelligence on an adversary’s defenses creates more value in intruding into adversary computer networks well in advance of any planned attack.

The second proposition concerns defense: Guarding against deceptive attacks may demand constant efforts to gain advanced knowledge of attackers’ capabilities. Because machine learning systems cannot simply be patched, they must be able to adapt to defend against deceptive attacks. Yet researchers have found that adaptations to defend against one form of deception are vulnerable to another form of deception. No defense has been found that can make a machine learning system robust to all possible attacks—and it is possible none will be found. Consequently, machine learning systems that adapt to better defend against one form of attack may be at constant risk of becoming vulnerable to another. In the face of an imminent threat by an adversary, the best defense may be to intrude into the adversary’s networks and gain information to harden the defense against their specific capabilities.

Together these two propositions suggest machine learning could amplify the most destabilizing dynamics already present in cyber competition. Whether attacking or defending, at the top tier of operations, machine learning attack vectors may create challenges best resolved by intruding into a competitor’s networks to acquire information in advance of an engagement. This would add to existing pressures on states to hack into their adversaries’ networks to create offensive options and protect critical systems against adversaries’ own capabilities. Yet the target of an intrusion may view the intrusion as an even greater threat—regardless of motive—if it could reveal information that compromised machine learning defenses. The already blurred line between offensive and defensive cyber operations may fade further. In a crisis, the potential for cyber operations to accelerate the path to conflict may rise. In peacetime, machine learning may fuel the steady escalation of cyber competition. Adversaries may adapt by targeting machine learning itself, including:

• Compromising supply chains or training processes to insert backdoors into machine learning systems that expose a potentially wide swath of applications to possible attacks.
• Poisoning training data, such as open source malware repositories, to degrade cybersecurity applications.
• Unleashing risky capabilities to circumvent defenses, such as malware with greater degrees of autonomy.
• Targeting defenders’ trust in machine learning systems, such as by inducing systems to generate “false positives” by mislabeling legitimate files as malware.

For the United States and its allies, harnessing machine learning for cybersecurity depends on anticipating and preparing for these potential changes to the threat landscape. If cyber defense increasingly relies on inherently flawed machine learning systems, frameworks and metrics will be needed to inform risk-based decisions about where and how to employ them. Securing the machine learning supply chain will demand collective governmental and private sector efforts. Finally, the United States and its allies must exercise caution in the conduct of their offensive operations and communicate with adversaries to clarify intentions and avoid escalation.