Researchers, companies, and policymakers have dedicated increasing attention to evaluating large language models (LLMs). This explainer covers why researchers are interested in evaluations, as well as some common evaluations and associated challenges. While evaluations can be helpful for monitoring progress, assessing risk, and determining whether to use a model for a specific purpose, they are still at a very early stage.

Now that the first set of milestones has passed for the Biden administration’s October 2023 executive order on artificial intelligence, where do things stand for biotech? This blog post gives an overview of the most recent action items, with a recap of expert commentary from CSET’s June 2024 Webinar on the AIxBio Policy Landscape.

The Biden Administration’s “Investing in America'' agenda designates $80 billion to focus on “place-based industrial policy,” representing a historic level of government investment. Policymakers across government need to understand how these investments help the United States to stay ahead of its primary strategic competitors, China and Russia. This post previews a CSET study that will explore how the creation and design of place-based innovation centers can enhance both U.S. economic and national security.

The National Artificial Intelligence Research Resource (NAIRR) pilot provides federal infrastructure, including computational resources, to U.S. AI researchers. This blog post estimates the compute provided through the pilot, and finds that it is roughly 3.77 exaFLOPS, the equivalent of approximately 5,000 H100 GPUs (using the tensor cores optimal for AI). The pilot is a significant first step in providing compute to under-resourced organizations, although it is a fraction of what is available to industry.

In their op-ed featured in The Wire China, CSET's Ngor Luong, Sam Bresnick, and Kathleen Curlee provide their expert analysis on the changing landscape for U.S. big tech companies in China.

CSET's Director of Strategy and Foundational Research Grants Helen Toner delivered a talk at TED2024 on the importance of developing smart AI policy, even in the face of uncertainty.

In her op-ed featured in War on the Rocks, CSET's Jenny Jun discussed the nuanced relationship between AI and cyber operations, highlighting both the optimism and caution within the U.S. government regarding AI's impact on cyber defense and offense.

In their op-ed featured in Breaking Defense, CSET's Sam Bresnick and Emelia Probasco provide their expert analysis on the involvement of US tech giants in conflicts, such as the Ukraine war, and raise important questions about their role and potential entanglements in future conflicts, particularly those involving Taiwan.

A recent topic of contention among artificial intelligence researchers has been whether large language models can exhibit unpredictable ("emergent") jumps in capability as they are scaled up. These arguments have found their way into policy circles and the popular press, often in simplified or distorted ways that have created confusion. This blog post explores the disagreements around emergence and their practical relevance for policy.

In his op-ed published by the Center for European Policy Analysis (CEPA), CSET’s Sam Bresnick shared his expert analysis on China's evolving military capabilities and its growing emphasis on battlefield information and the role of AI.