Executive Summary
For the last decade, breakthroughs in artificial intelligence (AI) have come like clockwork, driven to a significant extent by an exponentially growing demand for computing power (“compute” for short). One of the largest models, released in 2020, used 600,000 times more computing power than the noteworthy 2012 model that first popularized deep learning. In 2018, researchers at OpenAI highlighted this trend and attempted to quantify the rate of increase, but it is now clear this rate of growth cannot be sustained for long. In fact, the impending slowdown may have already begun.
Deep learning will soon face a slowdown in its ability to consume ever more compute for at least three reasons: (1) training is expensive; (2) there is a limited supply of AI chips; and (3) training extremely large models generates traffic jams across many processors that are difficult to manage. Experts may not agree about which of these is the most pressing, but it is almost certain that they cannot all be managed enough to maintain the last decade’s rate of growth in computing.
Progress towards increasingly powerful and generalizable AI is still possible, but it will require a partial re-orientation away from the dominant strategy of the past decade—more compute—towards other approaches. We find that improvements in hardware and algorithmic efficiency offer promise for continued advancement, even if they are unlikely to fully offset a slowdown in the growth of computing power usage. Additionally, researchers are likely to turn to approaches that are more focused on specific applications rather than the “brute-force” methods that undergirded much of the last decade of AI research. The release of AlphaFold, which made incredible progress on a long-standing problem in the field of biology without the need for record-breaking levels of computing power, may be an example of this new shift in focus.
These findings lead to a few recommendations for policymakers:
- If continued AI advancement relies increasingly on improved algorithms and hardware designs, then policy should focus on attracting, developing, and retaining more talented researchers rather than simply outspending rivals on computing power.
- As a specific example of the above, we suggest that institutions such as the National AI Research Resource should not view computing power alone as the primary way to support AI researchers. These institutions should also invest in providing researchers with the skills to innovate with contemporary AI algorithms and to manage modern AI infrastructure, or should actively promote interdisciplinary work between the AI field and other subfields of computer science.
- Finally, policymakers should take proactive steps to ensure that researchers with small or moderate budgets can effectively contribute to the AI research field. Concentrating state-of-the-art technologies among the small number of research centers possessing extremely large compute budgets risks creating oligopolistic markets and shrinking the talent pool and opportunities for researchers.