Other briefs in this series:

• Key Concepts in AI Safety: An Overview
• Key Concepts in AI Safety: Robustness and Adversarial Examples
• Key Concepts in AI Safety: Interpretability in Machine Learning
• Key Concepts in AI Safety: Reliable Uncertainty Quantification in Machine Learning

Introduction

Specification is the task of conveying to a machine learning system what exactly its designers would like it to do.¹ For some tasks—such as choosing which tiles in a CAPTCHA test contain a traffic light—it is relatively straightforward for the designer of such a system to write a precise description of what they are looking for. For many other tasks, however, it is difficult to capture the nuances of the intentions in precise, mathematical language.

In some ways, this type of challenge is not unique to machines. One prominent computer scientist describes specification problems in terms of familiar fictional analogues: As in the cases of King Midas, the Sorcerer’s Apprentice, or the genie in the lamp, you get exactly what you wished for, which may not necessarily be what you wanted.² Principal–agent problems in economics deal with related situations, where a task is delegated from one person (the principal) to another (the agent), but the agent’s understanding or incentives may diverge from the principal’s. Specification problems in machine learning systems arise due to similar dynamics: In all but the simplest settings, it is challenging to convey and incentivize desired behaviors, which may in turn lead to undesired behaviors. Ensuring that a given specification of a machine learning system results in a specific desired behavior and is in accordance with its designer’s intentions is a key challenge for machine learning research. To guard against failures, the goal is for machine learning systems to be robust to potential errors or inaccuracies in the specification. Getting this right will become increasingly critical as machine learning systems are deployed in higher-stakes and more complex settings. This brief provides an overview of specification problems for a policy audience, introducing key concepts, offering real-world examples, and suggesting implications for policymakers.

1. “Specification” can also be used to refer to other parts of the process of designing and training a machine learning system, such as model specification, but these are not included for the purposes of this paper.
2. Stuart Russell, “Of Myths and Moonshine,” Edge, November 2014, https://www.edge.org/conversation/the-myth-of-ai#26015.