Introduction

Given societal interests in the development and governance of artificial intelligence and machine learning systems, policymakers are keen to understand and track research and development on trustworthy AI topics. Tracking this topic is challenging, however, both because of the rapid developments in the field of AI/ML generally and also because trustworthy AI is a constantly developing, multifaceted concept that most scientific research papers address only in part or in a specific application.

In our previous Center for Security and Emerging Technology (CSET) report, The Inigo Montoya Problem for Trustworthy AI, we looked at how trustworthy AI keywords can (and cannot) be useful in identifying research papers related to trustworthy AI. In this follow-on analysis, we pair what we learned from our keyword approach with a different way of identifying areas of research: citations. Using CSET’s research clusters derived from CSET’s merged corpus of scholarly literature, we are able to contextualize the trustworthy AI keyword publications found in the prior report within the scientific research landscape. This approach of identifying research clusters with a high percentage of trustworthy AI keyword papers opens the aperture for finding trustworthy AI research, surfacing papers that do not use a trustworthy-AI term and also overcoming the problems that come with using generic keywords in searches, like safety and security. Overall, we found:

• The trustworthy AI terms reliability, safety, and robustness are widely dispersed across AI-research clusters, which further supports the idea from our previous research that these terms are widely studied and/or adopted metrics for AI systems.

• By contrast, we find the terms interpretability, transparency, explainability, security, and privacy are concentrated in fewer clusters, which may indicate that they are being studied as specific issues areas, rather than being broadly accepted and adopted characteristics across AI research.

• There are 18 clusters of AI-related research that are worth watching for those interested in the development of trustworthy AI. These 18 research clusters cover a broad spectrum of AI methods, techniques, and applications, including deep learning, adversarial attacks, word embeddings, image privacy, speech recognition, explainable AI, federated learning, algorithmic fairness, differential privacy, and robotics. For several of these clusters, the density of trustworthy AI keywords in the papers in the cluster is key to identifying the cluster as relevant to trustworthy AI research.