Key Takeaways

Different priorities can shape the risk-benefit calculus of contemporary open foundation models, in part because the implications of open models are often ambiguous or difficult to quantify. Faced with uncertainty, stakeholders will often lean on the benefits or risks associated with whatever issue area(s) they prioritize, be it research and innovation, safety and security, distribution of power in the artificial intelligence (AI) industry, or others. Below is an overview of the core benefits and risks of open models, per the different priorities covered in this blog.

The core potential benefits of contemporary open models stem from the fact that anyone can access, inspect, and alter their weights. 

• This accessibility can lower the barrier to entry for many actors to participate and collaborate in AI R&D, thereby accelerating R&D in certain areas and promoting the growth of an open AI ecosystem. 
• This ecosystem currently resides primarily within the United States and the territories of its allies, which may put them in a position to leverage it more effectively than adversaries, as well as allow domestic innovation to occur in more dynamic and customizable ways. 
• More actors may also have the opportunity to use AI and engage in R&D without relying on a handful of well-resourced technology companies, thereby reducing the likelihood (or extent) of inordinate concentration of power and control in the AI industry.
• Maintaining an open and relatively decentralized ecosystem may help ensure robust AI safety and security, as more actors can participate in the identification and remediation of vulnerabilities, biases, and other flaws in open models.

The core potential risks of contemporary open models also stem from the fact that anyone can access, inspect, or alter their weights. 

• The original developers cannot control who has access to the models, nor how they are used or altered, so U.S. adversaries and malicious non-state actors can leverage open models for their own ends and without the resources required to design models from scratch. 
• Open models may help adversaries keep pace with the U.S. in certain areas of AI R&D, which could influence strategic international competition to realize the economic and security benefits of AI. 
• More actors may also deploy and fine-tune open models for malicious purposes, such as scams, disinformation, non-consensual deepfake imagery, and supporting offensive cyber operations via spearphishing or social engineering. 
• Tracking the dissemination and misuse of open models is challenging because they are so easily accessible. Oversight will rely significantly on the transparency of downstream actors that use open models, but this includes nefarious actors that will not be transparent.

Further research is needed to inform the debate and empirically weigh the implications, both within and across priorities: to what extent do open models promote R&D, advantage U.S. competitors, enable versus complicate oversight, allow nefarious actors to misuse models maliciously (and for what purposes), or foster a more balanced distribution of power in industry. Answers to these questions will help policymakers better assess the implications of open models, and allow them to gauge the benefits and risks of newer AI capabilities as they emerge.

Introduction

In Support of Opening Models

“The short-term societal dangers of proprietary AI systems that will soon mediate everyone’s digital diet are considerably higher than any imagined catastrophe caused by the misuse of open source AI systems  . . . the benefits of open source AI platforms in terms of progress, safety, economic development, and cultural diversity are overwhelming.” – Yann Lecun

“Some lobbyists for large companies — some of which would prefer not to have to compete with open source — are trying to convince policy makers that AI is so dangerous, governments should require licenses for large AI models. If enacted, such regulation would impede open source development and dramatically slow down innovation.” – Andrew Ng

In Opposition to Opening Models

“While open-sourcing has historically provided substantial net benefits for most software and AI development processes, we argue that for some highly capable foundation models likely to be developed in the near future, open-sourcing may pose sufficiently extreme risks to outweigh the benefits.” – Centre for the Governance of AI

“I think the open-source movement has an important role in AI. With a technology that brings so many new capabilities, it’s important that no single entity acts as a gatekeeper to the technology’s use. However, as things stand today, unsecured AI poses an enormous risk that we are not yet able to contain.” – David Evan Harris

What are the implications of ‘open’ AI models, where their weights (i.e., the parameters that process inputs and generate outputs) are available online and can be downloaded, used, or fine-tuned by anyone with the necessary skills and resources? The answer is disputed, and it may appear that the discussion has coalesced around two broad camps, either for or against open models, as reflected in the quotes above. But a clash between two camps is an oversimplification of a much wider issue, involving a range of different perspectives and implications that should be considered.

The debate over open models has reached an inflection point following the October 2023 White House Executive Order on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. The order solicited input on the risks and benefits of dual-use foundation models that have “widely available” weights––and within 270 days, the U.S. Department of Commerce must submit a report to the president laying out the implications of open models and their recommended regulations. There is much uncertainty around the potential benefits and risks of open models, as well as how they will be developed, and how different actors will use them. Consequently, U.S. policymakers will likely be inundated with opinions, speculation, and regulatory proposals from a range of actors with their own priorities and interests.

As the Biden administration assesses the risks and benefits of AI, this blog post overviews the potential implications of not regulating open foundation models, with a focus on contemporary AI capabilities. This means maintaining the status quo, and avoiding any rules and restrictions around releasing model weights. It considers how different priorities can shape answers to the question of “given current AI capabilities, what might happen if the U.S. government left the open AI ecosystem unregulated?” By answering this question from different perspectives, it highlights the dangers of hastily subscribing to any particular course of action without weighing the potentially beneficial, risky, and ambiguous effects.

Terminology

Terms like open and closed, as applied to AI, have characteristics that vary from traditional definitions of open-source software and licensing. Different terms are sometimes used interchangeably without clear definitions, such as open models, open-source models, and open weights, resulting in confusion. In this section, we define ‘open’ and ‘closed’ models, and explain the spectrum of ‘openness’ on which a model falls based on the accessibility of its components and information about its development––all of which is (or is not) provided by the developers.

Open models have their weights available online, allowing anyone with sufficient resources and expertise to download and use them on their computing infrastructure and fine-tune them on their data. Weights are numerical parameters that a model learns during training; they enable its core functionality and allow it to map inputs to outputs (e.g., in the case of a large language model, prompting it with a question to generate an answer). Foundation models are expensive and difficult to develop, but if the weights are open and available, then anyone can obtain a model without needing the resources to develop it from scratch.

When people refer to open models, they are typically referring to open weights. However, models with publicly available weights fit along a spectrum of openness, and where they fit depends on the accessibility of their components (e.g., is the training data or surrounding code used to run or fine-tune the model readily available?), as well as the degree of transparency and documentation on how they were developed (e.g., how was the training data collected and processed? How was the model designed and evaluated?). 

All of these variables should be considered, as they can impact the ability to replicate and audit models, explain their outputs, scrutinize their functionality, and assess their capabilities and flaws. For example, if the goal is to scrutinize a model, then access to the training data may help explain its outputs; if the goal is to replicate a model, then access to most all of the components and development processes is typically necessary. More research is needed to gauge how different degrees of access and transparency can impact the ability to scrutinize or replicate open models. 

Lastly, open models come with licenses and stipulations from the original developers, which impact whether and how they can be commercialized, distributed, or used. There are ten criteria that determine if software is open-source, but open models can have licenses that diverge from those criteria (e.g., Meta’s LLama 2 models), and therefore should not be considered open-source. The term ‘open-source models’ should only be used in reference to models that meet those criteria.

Ultimately, the key variable that determines if a model is ‘open’ is the availability of the weights, even though other factors shape the degree of openness. It is because of this access to weights that an open AI ecosystem of users, researchers, and companies has formed, and a debate over the implications of this practice has increased.

Closed models do not have their weights available online, and the original developer controls their accessibility and functionality. This includes fully closed models that are only accessible to the developers, as well as those behind an API or web-based user interface (e.g., the models powering ChatGPT). Users can sometimes access these models over the internet, but their weights cannot be downloaded or moved to a different computing infrastructure. The original developer prohibits, mediates, or oversees the ability to fine-tune or alter the base functionality of closed models. Moreover, the ability to scrutinize or audit these models is often highly constrained, especially if the developers are not transparent.

Implications and Considerations

The implications of open foundation models are difficult to assess through a binary list of pros and cons. What is deemed ‘good’ or ‘bad’ can depend on what is prioritized. Different stakeholders have different priorities that shape what interests they seek to support, what concerns they seek to mitigate, and how they compare the potential benefits and risks of open models. Many of the implications are either ambiguous or difficult to quantify, and how stakeholders consider that uncertainty is often shaped by their priorities.

To illustrate the wide-ranging tradeoffs, this blog post considers five varying priorities that policymakers, researchers, and users may have. Because these priorities differ, they can shape the risk-benefit calculus of open models. While there is considerable overlap, these priorities are distinguished by their core interests, as shown below:

1. Research and Innovation
2. National Security and International Competition
3. Visibility and Oversight
4. Safety and Security 
5. Balanced Distribution of Power in Industry

People and organizations with different interests and concerns––over different timeframes––have different priorities. For some, safety concerns may supersede desires for innovation. They may foresee greater risk from open models because nefarious actors can access them, remove built-in safety features, and potentially misuse them for malicious purposes. Conversely, some who prioritize innovation may foresee greater benefits from open models because it may help democratize access and allow more actors to contribute to R&D. 

However, it is not always straightforward how certain interests are best achieved, and there can be significant differences in how people with similar priorities measure the potential benefits and risks. In contrast to the previous example, some who prioritize safety may argue that open models are beneficial because they allow more actors to scrutinize them and mitigate safety issues, while some who prioritize innovation argue that the impact of open models is exaggerated because many actors still lack the resources and access to effectively leverage them and contribute to R&D. 

Furthermore, some may seek to balance several priorities at once, and have different assumptions around whether those priorities are at odds. What if open models promote R&D, but also help adversarial nations keep pace in AI development? How should competing interests around R&D and international competition be weighed? What if some developers are highly transparent, but others are not? Should the U.S. promote visibility and transparency to improve oversight of open models, even though it may also help malicious actors find and exploit vulnerabilities? The answers to these questions largely depend on how any given stakeholder balances priorities and weighs the potential implications. 

Lastly, timeframes can significantly influence how one considers the potential risks and benefits of open models, regardless of what is prioritized. Given the rate of AI R&D, longer timeframes encompass a wider spectrum of potential AI capabilities. Some who favor opening today’s models may be opposed to opening future models that could pose greater risks. But the longer the timeframe, the more speculative things become, making it far more challenging to assess longer-term benefits or extreme risks stemming from the potential capabilities of models that have not been developed. 

This blog post focuses on the implications of contemporary AI systems and avoids considerations around future AI capabilities. It considers the landscape of viewpoints around current open models.

The remainder of this blog post assesses the potential implications of open models, as considered across five different priorities. Each section is structured as follows:

1. An overview of the priority and the interests it seeks to support.
2. A proposition summarizing the potentially beneficial, risky, and ambiguous implications of open models that are related to that priority.
3. A table assessing the implications in more detail.

The benefits and risks in each section are directly related to the priority, so anything that supports or threatens that priority is considered a benefit or a risk, respectively. Importantly, most of these implications are qualitatively different, so any section that lists more benefits or risks should not be interpreted as being more beneficial or risky. Lastly, many of the implications around open models also apply to closed models, to varying degrees, so any area that has significant overlap is marked with an alpha (ɑ) superscript.

– –

1. Those who prioritize research and innovation seek to improve AI performance, efficiency, and usability, expand its capabilities, and enable more engagement in AI R&D to accelerate technological progress. 

Proposition: Open models can be researched and developed by more actors, and innovation can occur in more dynamic, customized, and collaborative ways. However, many actors will still lack the resources, access, and incentives to effectively engage in open R&D.

Benefits to R&D	• Open sharing has helped drive much AI R&D, and the opening of models may help this trend continue and accelerate. • The barrier to entry in AI R&D may be lowered for less well-resourced actors who cannot develop expensive pre-trained models from scratch (e.g., academics and SMEs). • Platforms can emerge and act as open repositories for a global body of AI researchers and developers (e.g., Hugging Face and GitHub). Communities can form around these platforms, fostering cross-pollination of research and decentralized collaboration. • More actors can customize and fine-tune open models for specific applications and novel research, especially those with access to bespoke datasets that can be used to fine-tune models for unique tasks. This can be done on their own computing hardware and without restrictions.
Risks to R&D	• Actors may opt to use open models instead of paying for access to closed models, which may reduce the revenue of developers and disincentivize investments in capital-intensive AI R&D.  • Actors may avoid using, innovating upon, or investing in open models due to concerns over intellectual property and licensing constraints from the original developers.ɑ
Ambiguous Effects on R&D	• Openness and access across the open AI ecosystem vary, and the implications of this heterogeneity are unclear (e.g., whether or not open models with thorough documentation can benefit R&D more than open weights alone). Lower degrees of model access (e.g., to only the weights) may limit the ability to assess model capabilities and test how different training regimes impact model outputs, as well as increase redundancy in research. • Many actors lack resources, but it is unclear to what extent resource constraints may limit the benefits of open models for R&D. Actors may lack the data to fine-tune open models, as well as the resources to collect, refine, and process data for training. They may also lack the compute to use, fine-tune, or experiment with open models rigorously and at scale.

2. Those who prioritize national security and international competition are concerned about strategic technological rivalry with adversarial states, including the race to realize the economic benefits of AI, as well as over AI systems that could support national security interests. They seek to foster national AI R&D while minimizing that of adversaries.

Proposition: Open models are available to U.S. constituents, partners, competitors, and adversaries. This may help both friends and foes keep pace in AI R&D and allow any nation to leverage the software for its own ends, but the extent to which it helps them is unclear.

Benefits to National Security	• Much of the open-source software ecosystem resides within the United States and the territories of its allies, as do many of the organizations engaging in open AI R&D. This may help the United States leverage open innovation (to support its security and economic interests) more effectively than adversaries, as well as put it in a better position to shape international norms around AI. • An open, U.S.-promoted AI ecosystem may help foster multilateral collaboration on AI policy and security issues, as it may create a common and more transparent baseline from which all states can assess the technology.
Risks to National Security	• It will be difficult to limit the dissemination of open models that may be valuable to adversaries. Once opened, a model cannot later be closed or effectively contained. Adversaries can access, use, and adapt open models that are released by U.S. actors (e.g., Meta has released several models and will likely continue to do so). • China’s ability to develop frontier models to support their national security interests may not be as inhibited by U.S. compute-based export controls because open models are already pre-trained, they are often small and easy to use, and fine-tuning them requires far less compute than the initial training. It may also be difficult to assess if and how open models could be valuable to an adversary, given their dual-use characteristics and customizability.
Ambiguous Effects on National Security	• Foreign actors are also releasing capable open base models (e.g., China’s DeepSeek, the UAE’s TII, and France’s Mistral AI), and it is unclear the extent to which their ability to foster their own indigenous open AI ecosystems is aided by U.S. actors opening models. • The U.S., its partners, and its adversaries may deem open models too unreliable for sensitive use cases (e.g., military or intelligence). They may instead partner with developers and use closed models to have more control over the pre-training, which would complicate the impact of open models on international security dynamics.

3. Those who prioritize visibility and oversight seek to measure AI use, research, development, testing, and evaluation (RDT&E). This includes assessing the processes, people, and politics behind it, tracking the dissemination of models, and promoting transparency around AI development.

Proposition: Open models can both complicate and promote oversight. This depends on the transparency of developers and users, the openness and accessibility of models, the scale of model dissemination, and the degree to which open RDT&E is visible on common platforms.

Benefits to Visibility	• Many actors are transparent about how they develop and use open models, often more so than developers of closed models. Many support initiatives around transparency and openness (e.g., EleutherAI, Nomic AI, and Petuum), which can help visibility and facilitate oversight. • Platforms such as Hugging Face and GitHub can help aggregate and track a significant portion of open models, who develops them, how they are developed, and when new fine-tuned versions are released. • A handful of well-resourced actors (e.g., Meta, Mistral AI, and Microsoft) have developed many of the popular open base models, so oversight initiatives focusing on these few companies and their base models are relatively feasible.ɑ
Risks to Visibility	• Developers of open models cannot oversee how they are used or fine-tuned, so the onus for transparency will be on downstream actors. Moreover, open fine-tuned models are often less rigorously evaluated than the original open base models, which can make visibility and oversight more challenging. • It may be more difficult to track and oversee a distributed body of open models, developers, and users. Moreover, actors may privately use and fine-tune open models, further inhibiting oversight. • Actors may not be transparent about how they develop, release, fine-tune, or alter open models. Moreover, government-led efforts to foster transparency norms (and voluntary commitments) with domestic actors can be challenging, and even more so internationally.
Ambiguous Effects on Visibility	• Transparency around a base model shapes the transparency of downstream models based on it (e.g., not knowing a base model’s training data inhibits insight into any fine-tuned version).ɑ The impact of this on visibility and oversight across many models in the open AI ecosystem is unclear. • The degree to which developers will be transparent about their model development, training, and fine-tuning processes is unclear; some will be highly transparent while others will not.ɑ

4. Those who prioritize safety and security seek to examine and assess AI for potential risks and vulnerabilities. This includes assessing the potential for misuse and whether AI can be used safely and reliably for various applications, as well as evaluating the security of the models themselves (and the systems in which they are embedded).

Proposition: Open models can be examined and used by anyone. More actors can find and fix vulnerabilities, or find and address safety issues. However, more actors can exploit discovered vulnerabilities, remove safety features, have lax or non-existent security standards, or otherwise misuse or maliciously deploy models.

Benefits to Safety & Security	• More actors can scrutinize open models, identify vulnerabilities and safety issues, and implement patches and safety measures. There is a long history of open-sourcing software to crowdsource vulnerability discovery, which typically makes the software more secure than if it was only assessed in-house. • More actors can assess open model functionality, limitations, and biases. Greater access often improves auditability, which may bolster the reliability and trustworthiness of downstream AI applications.
Risks to Safety & Security	• Nefarious actors can use open models maliciously, such as for disinformation, scamming, producing non-consensual deepfake imagery, and supporting offensive cyber operations via spear phishing, reconnaissance, and social engineering (whereas developers of closed models have more means to identify and disrupt this malicious use). They can also exploit vulnerabilities in open models, and extract data from them. • The developers of open models cannot monitor or control how they are used or fine-tuned. Licenses are the primary mechanism for disincentivizing misuse, but some actors may be willing to infringe on them. • Unknown or unresolved vulnerabilities, biases, and other flaws in open base models can propagate downstream to fine-tuned models.ɑ But even if issues in the base model are identified and resolved, downstream users and developers may not decide (or be able) to implement fixes to their fine-tuned models. • Fewer safeguards may be applied to open models, and it may be difficult to foster safety and security standards across many users and developers. • Actors can remove safety features built into open models. Although safety features can also be removed from closed models via fine-tuning APIs, the problem is more acute with open models because they (and fine-tuned versions of them) are not controlled by the original developer. • The likelihood of unintended misuse may increase, especially if open models lack transparency around their limitations and intended use cases. Moreover, fine-tuning can unintentionally degrade built-in safety measures.
Ambiguous Effects on Safety & Security	• It is unclear if and how actors will adopt safety measures when deploying open models, as well as how difficult it will be for them to adopt such measures.ɑ  • The breadth of potential users and depth of potential use cases increases the opportunities for legitimate use, misuse, and malicious use. Some research has considered marginal risks from open models, but the overall implications of this dynamic are unclear. • The balance between offense and defense in AI is unclear (e.g., finding and exploiting vulnerabilities versus finding and patching them).ɑ This dynamic is loosely analogous to open-source software, but patching AI vulnerabilities can be more difficult than patching traditional software vulnerabilities.

5. Those who prioritize a more balanced distribution of power in industry are concerned about AI R&D and commercialization, but focus on who can engage in it, how they can engage, and how that can translate to excessive political or economic influence within the hands of a few organizations.

Proposition: Open models can allow more actors to use AI and engage in R&D without relying on closed models that are controlled by the developers. However, open weights alone do not guarantee a more balanced distribution of power, and established companies may leverage the open AI ecosystem to further entrench their positions.

Benefits to Balanced Power Distribution	• More actors can use open models without restriction or control from the original developers. Only a few companies currently have the resources to develop capable base models from scratch (e.g., OpenAI, Anthropic, and Google), but once a model is open, the downstream control over its use is distributed. • More academics, SMEs, non-profits, and other less well-resourced actors can more effectively engage in AI R&D. This wider engagement may reduce the concentration of cutting-edge research among a few companies. Over time, this may allow more entrants into the field, increase competition, and improve the market dynamics of the AI industry.
Risks to Balanced Power Distribution	• Few companies have the resources to develop large open base models, deploy them at scale, and financially support strategies of openness that may not generate revenue in the short term.ɑ This may help them shape AI development, leverage open models for specific use cases, implement stringent licensing requirements on models they release, benefit from partnerships with open platforms to supply resources, and further concentrate their influence over the industry • Smaller organizations seeking to develop models from scratch may still face a significant barrier to entry, particularly due to limited access to compute, data, and talent.ɑ Therefore, many actors will rely on the open models released by more established companies, which may further entrench their influence. • Companies that develop models from scratch may still retain much of the key knowledge and expertise, even if weights are released. Open weights alone do not allow model replication or thorough scrutinization, and much of the RDT&E done by the original developers is not transparent and cannot be examined by outside actors.
Ambiguous Effects on Balanced Power Distribution	• It is unclear how different licenses (with varying stipulations) for different open models help established companies maintain or entrench their market positions. Moreover, it is unclear how established companies leverage open models that are released by other established companies. • It is unclear the extent to which frontier AI will remain in the hands of a few companies that can afford to develop the most expensive and capable models. At some point, there may be diminishing returns to scaling to ever larger models. This may allow open models to close the gap near the frontier if there is a discernible slowdown. It is also unclear if and when well-resourced companies will open the weights of state-of-the-art models.

Recommendations

Balancing sometimes conflicting priorities is incredibly difficult, and tradeoffs are inevitable. To assist in this analysis, the following recommendations can help better assess the benefits and risks of open foundation models: 

1. Ambiguity is a common thread across all the priorities. NTIA and NIST should conduct research and solicit stakeholder feedback to reduce this ambiguity (to the extent that it is possible) and have a clearer understanding of the overall implications. This can include research into: 
   • How resource constraints and different degrees of model access/openness impact the benefits of open models for R&D. How truly beneficial are open models to AI R&D, and what specific types of research have open models enabled that closed models have not?
   • The extent to which open models, either released by U.S. or foreign actors, help adversaries keep pace in AI R&D. To the degree that it helps them keep pace, does this actually help them leverage AI to support their national security interests, and does this translate to a significant national security concern for the United States?
   • Assessing the degree of visibility into the overall open AI ecosystem, the extent to which that visibility enables or complicates oversight, and the mechanisms through which the government can promote visibility and transparency.
   • The extent to which open model developers (of either base models or fine-tuned versions) are implementing or adopting safety measures, and how the removal of those features may impact the risks of misuse. Moreover, how the degree of visibility into the open AI ecosystem impacts the ability to even assess these safety measures (or lack thereof).
   • The extent to which established companies can and cannot leverage the open AI ecosystem for their own ends, and how that may or may not translate to further concentration of power in industry. Moreover, the varying characteristics of open models (i.e., models with different licenses and resource requirements) can have different impacts on this dynamic.
2. There may be policies that help reduce risk without undermining benefits, particularly regarding visibility and oversight. For example, implementing non-burdensome transparency requirements on organizations that open their foundation models, such as the disclosure of general information on a model’s design, the resources used to train it, and its intended use. This increased visibility may help broader risk-benefit assessments across other priorities.
3. Over time, open model performance will likely improve, and new capabilities and use cases will likely emerge. Policymakers should consider how new, specific AI capabilities may change the risk-benefit calculus across different priorities. But avoid overweighting hypothetical risks from unrealized AI capabilities, as actions that stem from these concerns may unintentionally increase other risks (e.g., increasing the concentration of power in industry, stymieing innovation, or threatening the domestic open AI ecosystem to a degree that pushes developers abroad).
4. Do not assume that the performance and capabilities of open models will always lag behind those of closed models to a significant degree. Over time, the resources needed to develop capable models may be reduced due to improvements in algorithmic and computational efficiency, as well as improvements in training data quality. Moreover, companies may decide to open the weights of frontier models, thereby reducing the performance gap between open models and closed state-of-the-art models. Therefore, policymakers should consider how the risk-benefit calculus could change if frontier models are open and accessible.

Thanks to John Bansemer, Jenny Jun, Josh Goldstein, Thomas Woodside, Jessica Ji, Helen Toner, and Chris Rohlf for their invaluable feedback on this post.