Summary

The integration of artificial intelligence into military operations has become a significant focus for armed forces globally. Military commanders are interested in AI’s potential to improve decision-making, especially at the operational level of war, where they must integrate a lot of information quickly to make life-and-death decisions. However, the enthusiasm for AI-enabled decision support systems (DSS) must be balanced with an understanding of their capabilities and limitations to ensure appropriate and effective deployment. This report reviews recently proposed uses of AI-enabled DSS, provides a simplified framework for considering AI-DSS capabilities and limitations, and recommends practical risk mitigations that commanders might employ when operating with an AI-enabled DSS. Our framework for considering AI-DSS intended for operational decision-making emphasizes three critical areas.

1. Scope considerations: Is the scope of the AI-DSS well-defined and understood?
   • Context shifts. AI systems are prone to fail if used in settings that are meaningfully different from their training data.
   • Projection and prediction. There is an important difference between predictions based on physical laws and those involving human interactions. For the latter, we lack accurate models and directly observed data.
   • Flexible or unclearly scoped systems. AI-DSS can be flexible, but without well-defined use cases or guardrails, they can confuse operators and lead to misuse.
   • Irreducible uncertainty. Questions like “What will the enemy do?” have an inherent level of uncertainty that cannot be fully eliminated. DSS users must understand that certainty is an impossible goal and human judgment is still required when using AI-DSS.
2. Data considerations: Does the training data substantiate the AI-DSS’ conclusions?
   • Quality and fidelity. High-quality, relevant data is critical for effective AI systems but challenging to gather and maintain. Human behavior data is particularly challenging to use effectively due to its indirect observability and demographic variability.
   • Skewed data. Military commanders struggle to obtain accurate data on friendly and enemy forces. Data biases—such as those arising from sensor availability, deception, or in social media—can significantly impact AI system outputs.
   • Scarce data. The ability of an AI system to provide analysis or predict outcomes in combat or war may be limited because data about combat and war is limited. Traditional intelligence methods, which can combine insights and inferences that rely on a richer understanding of the relevant context, can be more valuable.
3. Human-machine interaction: What are the capabilities and limitations of the human-machine team as a single system within a given context?
   • False expectations with LLMs. Large language models (LLMs) are powerful tools, but they must be applied with care as they can mislead users by confidently presenting incorrect information, fabricating justifications, or increasing user acceptance of erroneous recommendations.
   • Human biases. Users must understand how their cognitive biases—such as automation bias, confirmation bias, or recency bias—may be affected by AI-DSS outputs, especially in stressful scenarios.
   • Organizational biases. Overreliance on DSS due to perceived ease of use can lead to poor decision-making, especially in extreme situations when risk tolerance is high. Organizations must be careful to avoid hasty or under-resourced decision-making based on a false perception of AI capabilities.

Based on our analysis, we recommend the following risk mitigation strategies when using AI-DSS:

1. Set context- and risk-based criteria for deployment: Commanders should set the time, place, and context for authorizing DSS use and prepare forces to adapt software settings as conditions and risk tolerance change.
2. Train and qualify AI-enabled DSS operators: Operators should be thoroughly trained on DSS capabilities and limitations. Those involved in lethal operations should undergo examinations for official qualifications appropriate to their role in operating the system.
3. Establish a continuous certification cycle: Units leveraging AI-DSS should be regularly certified to reduce the risk of inappropriately deploying or operating the system. Sharing performance metrics with data scientists, operations analysts, and experts in continuous tests and evaluations can help validate continued responsible use of AI-DSS and also support technical evolutions.
4. Designate a Responsible AI Officer: Akin to establishing safety and mishap programs, responsible AI (RAI) officers in military units can serve as local conduits for new information, promoting broad-based AI literacy, reporting AI incidents or mishaps to a higher authority, and mitigating AI-DSS risks.
5. Document incidents and harms: Documenting AI system flaws and user mistakes is essential for avoiding repeat errors and for building trust through transparency. RAI officers should be responsible for such documentation, akin to mishap reporting processes already in place in the services. The integration of AI into military decision-making presents both opportunities and challenges. By carefully considering the scope, data quality, and human-machine interaction, and by implementing rigorous training, certification, and safety measures, military organizations can leverage AI more effectively while mitigating potential risks.