Assessment

CSET Senior Fellow Dr. Heather Frase discussed her research on effectively evaluating and assessing AI systems across a broad range of applications.

Lessons From the Ukraine-Russia War

Issues in Science and Technology
| Spring 2023

CSET’s Jaret C. Riddick and Cole McFaul shared their expert analysis in an article published by Issues in Science and Technology.

The case for slowing down AI

Vox
| March 20, 2023

CSET'S Helen Toner was recently cited in a Vox piece discussing the popular narrative of an AI arms race between the US and China.

Militaries seek to harness artificial intelligence for decision advantage. Yet AI systems introduce a new source of uncertainty in the likelihood of technical failures. Such failures could interact with strategic and human factors in ways that lead to miscalculation and escalation in a crisis or conflict. Harnessing AI effectively requires managing these risk trade-offs by reducing the likelihood, and containing the consequences of, AI failures.

One Size Does Not Fit All

Heather Frase
| February 2023

Artificial intelligence is so diverse in its range that no simple one-size-fits-all assessment approach can be adequately applied to it. AI systems have a wide variety of functionality, capabilities, and outputs. They are also created using different tools, data modalities, and resources, which adds to the diversity of their assessment. Thus, a collection of approaches and processes is needed to cover a wide range of AI products, tools, services, and resources.

ChatGPT grabs headlines but Chinese competitor to face censorship

South China Morning Post
| February 20, 2023

The South China Morning Post quoted Dahlia Peterson and Hanna Dohmen, both research analysts at CSET, in an article about China's struggles in developing an equivalent of ChatGPT.

Comment to NIST on the AI Risk Management Framework

Mina Narayanan
| September 29, 2022

CSET submitted the following comment in response to the National Institute for Standards and Technology's second draft of its AI Risk Management Framework.

Exploring Clusters of Research in Three Areas of AI Safety

Helen Toner and Ashwin Acharya
| February 2022

Problems of AI safety are the subject of increasing interest for engineers and policymakers alike. This brief uses the CSET Map of Science to investigate how research into three areas of AI safety — robustness, interpretability and reward learning — is progressing. It identifies eight research clusters that contain a significant amount of research relating to these three areas and describes trends and key papers for each of them.

Classifying AI Systems

Catherine Aiken and Brian Dunn
| December 2021

​​This Classifying AI Systems Interactive presents several AI system classification frameworks developed to distill AI systems into concise, comparable and policy-relevant dimensions. It provides key takeaways and framework-specific results from CSET’s analysis of more than 1,800 system classifications done by survey respondents using the frameworks. You can explore the frameworks and example AI systems used in the survey, and even take the survey.

Key Concepts in AI Safety: Specification in Machine Learning

Tim G. J. Rudner and Helen Toner
| December 2021

This paper is the fourth installment in a series on “AI safety,” an area of machine learning research that aims to identify causes of unintended behavior in machine learning systems and develop tools to ensure these systems work safely and reliably. The first paper in the series, “Key Concepts in AI Safety: An Overview,” outlined three categories of AI safety issues—problems of robustness, assurance, and specification—and the subsequent two papers described problems of robustness and assurance, respectively. This paper introduces specification as a key element in designing modern machine learning systems that operate as intended.