CSET

Translating AI Risk Management Into Practice

Mina Narayanan

and Heather Frase

June 26, 2023

CSET's AI Assessment team provides a template that helps organizations create profiles to guide the management and deployment of AI systems in line with NIST's AI Risk Management Framework.

Executive Summary

Profiles provide detailed guidance for managing risk related to AI systems, and serve as a valuable addition to the National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework (NIST AI RMF). We present a template that helps organizations create profiles that offer actionable guidance for managing AI systems throughout their lifecycle. The template encourages defining roles for managing risk, clarifying the actions assigned to each role, and noting when roles are most relevant throughout an AI system’s lifecycle. Organizations that implement this template will be able to more easily translate the NIST AI RMF into practice.

The NIST Framework and its Functions

The National Institute of Standards and Technology (NIST) has developed an AI Risk Management Framework (RMF) that provides risk management guidance for people building or using AI systems.1

The AI RMF is structured around functions. Functions organize AI risk management activities at their highest level, where each function is composed of categories, or actions that achieve functions. Categories are then broken into subcategories, which are the outcomes of each action.

More specifically, the framework is organized around four functions: Govern, Map, Measure, and Manage. The Govern function underwrites all other functions. It is the practice of establishing policies, procedures, norms, accountability structures, and resourcing to facilitate a culture that appropriately considers organizational risk. Map provides context that informs how risk from AI is measured and managed. Measure analyzes, assesses, benchmarks, and monitors AI risk. It can provide a justification for making tradeoffs between trustworthy characteristics using qualitative, quantitative, or mixed methods. Finally, Manage involves prioritizing and treating risk. Figure 1 illustrates how a function is disaggregated into a category and its subcategories.

Figure 1: Disaggregation of Map function into a category and its subcategories

Figure 1: Disaggregation of Map function into a category and its subcategories. Map (Function), Map 4 (Category): Risks and benefits are mapped for all components of the AI system including third-party software and data. Map 4.1 (Subcategory): Approaches for mapping AI technology and legal risks of its components – including the use of third-party data or software – are in place, followed, and documented, as are risks of infringement of a third party’s intellectual property or other rights. Map 4.2 (Subcategory): Internal risk controls for components of the AI system, including third-party AI technologies, are identified and documented.
Source: The National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0).2

The functions, categories, and subcategories are not necessarily linked to a single stage of an AI system’s lifecycle. In fact, they can be implemented throughout multiple lifecycle stages. For example, performing AI impact assessments, an action that falls under the Measure function, can occur while a system is being designed, trained, or used in an operational environment. Similarly, the guidance that “Decision-making related to mapping, measuring, and managing AI risks throughout the lifecycle is informed by a diverse team” under the Govern function remains relevant throughout the lifetime of an AI system. 

NIST AI RMF Profiles

NIST has proposed the creation of profiles, or instantiations of the AI RMF for different end uses.3

Profiles can vary based on factors such as risk tolerance or resource availability, and their content may be granular or high-level. The focus of a profile may not even be an AI system – it can be activities or processes that are common across multiple sectors, such as acquisition approaches. The coverage and detail of profiles will also likely change as time goes on and AI risk management matures. 
CSET is introducing a template (available to download below) for one type of profile, namely profiles that offer actionable guidance for managing AI systems throughout their lifecycle. However, this template could be adapted to fit the needs of other types of profiles. The following content was informed by conversations with representatives from government agencies and an international organization, as well as researchers from an academic research center, public policy center, and non-profit organization. The opinions expressed in the template are those of CSET and do not necessarily reflect the views of consulted individuals and organizations. The structure of cybersecurity risk management profiles4 and the Test & Evaluation Master Plan written by the Director, Operational Test & Evaluation5 at the Department of Defense have also influenced the template’s layout.

Download Profile Template
NIST AI RMF Profile Template
  1. The National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (Washington, D.C.: Department of Commerce, 2023),https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf.
  2. The National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (Washington, D.C.: Department of Commerce, 2023), 27, https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf.
  3. The National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (Washington, D.C.: Department of Commerce, 2023), 33, https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf.
  4. The National Institute of Standards and Technology, Cybersecurity Framework: Examples of Framework Profiles (Washington, D.C.: Department of Commerce, 2023), https://www.nist.gov/cyberframework/examples-framework-profiles.
  5. The Office of the Director, Operational Test and Evaluation, Director, Operational Test and Evaluation (DOT&E) Test and Evaluation Master Plan (TEMP) Guidebook (Washington, D.C.: Department of Defense, 2017), https://www.dote.osd.mil/Portals/97/docs/TEMPGuide/TEMP_Guidebook_3.1a.pdf?ver=2020-02-27-131746-600