Executive Summary

When a software developer, government, or independent researcher discovers a cyber vulnerability, they can turn to a broad set of policies, procedures, and informal norms governing responsible disclosure. These practices have evolved over the last 40 years and, in many cases, have helped shorten the period between vulnerability discovery and remediation. As machine learning applications become more widespread, attention has turned to their potential vulnerabilities. While there are relevant analogues to cyber vulnerability discovery and disclosure, ML vulnerabilities are often fundamentally different from their cyber counterparts. It is too early to know whether these ML vulnerabilities will be as impactful as their cyber predecessors, but it is already becoming clear that some of the practices and policies surrounding vulnerability disclosure and management must adapt.

ML models are shaped by data rather than the logic of a programmer, so ML vulnerabilities are often harder to identify precisely, but they can be easy to exploit. Some attacks on ML include altering inputs to cause an otherwise properly performing system to err, tampering with the underlying ML model or its training data, stealing the ML model or training data, and exploiting bugs in the ML software that grant access to the computers running it.¹

By considering how a range of vulnerability disclosure policies apply to a set of prototypical ML vulnerabilities, we identified four ways that ML vulnerabilities tend to differ from traditional cyber ones:

1. Patching may be more difficult for ML vulnerabilities: Patches for ML vulnerabilities often only partially close the security hole while incurring significant expense or performance penalties. Because of these challenges, disclosures should emphasize mitigations over patches.
2. Many ML systems may be inherently vulnerable: Because vendors may be unable or unwilling to patch ML systems, they are often inherently vulnerable. When vendors no longer support legacy software, such as older Windows XP systems, there are greater risks that demand more attention from users—the same is true for ML systems.
3. ML vulnerabilities can be specific to each user: ML systems are often tailored to each user and adapt over time. Because of this feature, a vulnerability might be unique to one user’s model and not apply to other users even if they all used the same base model. Because each model is slightly different, it will be harder to verify whether a base model, or one built from it, is secure.
4. Proof-of-concept exploits for ML are less practically useful: ML exploits tend to be brittle and have high failure rates. To date, they have not been as disruptive in the real world as they appear to be in the academic or laboratory setting. They tend to be more useful as warnings to improve security than as viable threats.

These differences lead to several conclusions for vulnerability discoverers, authors of disclosure guidance, and the ML security community writ large:

Findings:

• Expect slower and imperfect patches and fixes. The broader security community should adjust expectations away from a simple process that delivers effective fixes with little performance cost.
• Expect a shift in responsibility for mitigation and remediation toward users and away from vendors. Vendors’ inability to secure their systems means that more security decisions must be made by users or their organizational security teams.

Recommendations:

• Improve risk assessment for ML applications. The security decisions that users or their security teams make will involve difficult tradeoffs. They will require a deeper understanding of ML vulnerabilities, the actors who might exploit them, and their impacts.
• Focus on developing broader mitigation strategies beyond patching. The shift toward mitigations from patches will require the security community to place more emphasis on developing these defenses and techniques. This reprioritization can better inform users by leading to disclosures that might otherwise go unreported. It also justifies a broader defense-in-depth approach for ML that emphasizes resiliency.
• Inform and empower users and security managers rather than just vendors when disclosing vulnerabilities. Presently, vulnerability disclosures are primarily aimed at informing vendors but should focus more on informing users of ML systems.
• Prioritize proof-of-concept exploits. For traditional software, defenders can determine the presence or extent of vulnerabilities without exploiting them. That seems less true for ML systems, which are more likely to be uniquely tailored to their users. ML exploits also currently tend to ignore real world challenges like gaining access to conduct attacks or the effect of varied lighting conditions, noisy data transmission, or redundant sensors. Together, these factors imply that disclosures should make a greater effort to provide exploits for ML vulnerabilities.

1. Ram Shankar Siva Kumar et al., “Failure Modes in Machine Learning,” Microsoft, December 1, 2021, https://docs.microsoft.com/en-us/security/engineering/failure-modes-in-machine-learning; Andrew Lohn, “Hacking AI: A Primer for Policymakers on Machine Learning Cybersecurity” (Center for Security and Emerging Technology, December 2020), https://cset.georgetown.edu/research/hacking-ai/.