Building standards for the next phase of AI

Our mission is to ensure that artificial general intelligence benefits all of humanity. As outlined recently by Sam Altman and Jakub Pachocki, we prioritize our work towards this mission with three main goals:

1. Navigate the next period of AI progress, by building an automated AI researcher, iterating with it on the alignment problem, and finding ways for people to remain part of the self-improvement loop.

2. Deliver the benefits of scientific progress and economic growth that very intelligent machines enable.

3. Empower everyone individually with a personal AGI.

AI is advancing across many companies and countries, and progress across the field changes the opportunities and challenges facing everyone. Alongside its other benefits, capable, aligned AI will be an important part of how we navigate this transition—strengthening defenses, advancing alignment research, and helping people understand and guide the next stages of AI development.

Navigating this transition safely requires alignment research to keep pace with these capabilities so that the systems we and others build remain aligned with human values and under human control. This becomes increasingly important as AI systems grow more capable and autonomous, including as they take on more of AI research and development itself. Making continued AI progress safe and beneficial will require both advances in alignment research and shared standards to guide development across labs and countries.

Automated AI research can involve varying degrees of human supervision. As AI systems take on more of the work of developing successive generations of AI, they can increasingly drive a process of recursive self-improvement (RSI), even while people remain involved. As this process becomes more automated, the pace of AI progress could accelerate rapidly.

We believe automated AI research will yield models that directly enhance people’s lives. It can bring down the cost of advanced intelligence so that people worldwide can benefit. Automated research could also help us substantially improve alignment and build defenses against increasingly capable AI—an automated AI researcher can also be an automated AI safety researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures.

Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely. Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices⁠(opens in a new window) about the benefits and risks. Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand. From here, AI could become more dangerous, less aligned, and, on the whole, a danger to people. The Hugging Face Incident we disclosed, while not a direct result of RSI, is a preview of the kinds of risks that could become much more severe without robust safeguards and alignment.

International standards for safety and security practices in frontier AI development may be as important to pacing the frontier as alignment research itself. Standards can create shared definitions of high-quality evidence and agreed-upon baselines for the rigor of technical safeguards. In short, they help us answer the question, “What does good look like in the mitigation of catastrophic AI risk?”

Existing democratic institutions like the Center for AI Standards and Innovation (CAISI), state laws, and a federal framework for AI can all help with this, as well as new forms of public-private partnerships⁠(opens in a new window). But AI development and deployment is happening in multiple countries, adoption is occurring globally, and its impacts will likely be experienced by everyone in some way on some timeline.

International efforts to develop standards are thus needed to solve some key challenges:

• Fragmentation—Evaluations, reporting requirements, and incident definitions by different nations could conflict, making it harder to compare evidence, understand emerging capabilities, and respond to risks that cross borders.

• Collective action—Each nation acting independently can produce outcomes that no nation wants. RSI has the ability to accelerate AI research itself, potentially beyond our collective ability to understand progress, assess risks, and maintain meaningful human oversight.

• Uneven capacity—The activity of frontier development and related expertise is not evenly distributed across the world. This accentuates both of the problems above.

These challenges apply to both open and closed models.

To be clear, any AI lab that pursues automated AI research or other advanced capabilities must take accountability for doing so safely, in accordance with basic principles of self-responsibility and existing laws. Our rationale for standards is rooted in avoiding the concentration of power, and producing better practical outcomes. Standards provide a way for more stakeholders outside of the labs to have a say in how this technology should unfold, and a visible set of principles that can be relied on irrespective of the specific practices of any particular lab. And we will likely get better results if parties collaborate to address the challenges above.

That is why we believe the United States should lead an effort to work together with countries around the world to develop global technical standards for frontier AI, including for RSI.

While we expect this concept to evolve substantially over time, two aspects will be essential.

One way of accomplishing this would be to leverage the emerging network of AI safety institutes—such as those already established in Australia, Canada, Germany, France, Kenya, Japan, Korea, Singapore, India, and the United Kingdom—to facilitate standard setting through the CAISI and national industry bodies, with a specific focus on (1) frontier AI models and developers, as measured by capability benchmarks; and (2) benefit-risk management for automated AI research, including RSI. The United States can build upon CAISI’s creation of the International Network for Advanced AI Measurement, Evaluation, and Science⁠(opens in a new window) in 2024, bringing together public institutions to cooperate on AI measurement, evaluation, and science.

Standards developed by this effort would provide a common technical foundation for capability measurement and evaluation, risk assessment, and safeguard sufficiency. These technical standards would not be licenses, mandatory prerelease review, or approval requirements for AI models. National governments would decide whether and how to incorporate these standards into their own legal systems.

This work should also support a competitive AI ecosystem by consulting open model and closed model developers, independent technical experts, and academia. Common standards should be developed transparently and designed so that they do not advantage particular companies, countries, or business models, including by making it harder for new entrants or open-weight developers to compete.

As we have suggested⁠(opens in a new window), this approach can draw lessons from areas such as aviation and financial stability, where countries have developed common technical standards and trusted channels for cooperation without giving up national authority. It should also work closely with established and newer standards bodies such as ISO⁠(opens in a new window), the Frontier Model Forum⁠(opens in a new window), Agentic AI Foundation⁠(opens in a new window), and the Open Secure AI Alliance⁠(opens in a new window), as well as implementation focused organizations, like the Appia Foundation, which is developing practical specifications that connect international standards with real-world AI assessments.

International AI standards will be particularly crucial in managing increasing autonomy in AI research and recursive self-improvement. Specifically, these could include standards for:

• Evaluation of RSI-relevant AI progress and the amount of autonomous research happening within an AI company. Our recent report on research acceleration is an initial contribution to this effort.

• Human oversight over automated AI research, including what kinds of automated AI research processes should trigger immediate human review.

• Incident classification, tracking, reporting, and responding for alignment and automated AI research issues, such as common incident severity levels and reporting thresholds. Our misalignment reporting framework is an early contribution.

In addition to these standards, we believe it will be important for critical infrastructure operators and governments worldwide to establish secure channels of communication to share national security concerns, emerging vulnerabilities and threats, and best practices in the fast-evolving field of frontier AI safety. Dialogue between the United States and China in these areas would be a positive step, and upcoming talks are happening at an opportune time.

Pacing AI development is not about maintaining a predetermined speed. Technically, it is about ensuring that alignment research and deployment of that research stay ahead of capabilities. Societally, it is about the use of standards, institutions, and collaboration to amplify networks of industry, societal relationships, and market forces to produce positive sum outcomes. The recent call for collective action on cyber defense is an example of the latter.

The United States is well positioned to lead because its AI industry is at the technical frontier, and it still stands in a privileged global network position in critical areas such as finance, trade, defense, technology, and information systems. Leading now will determine whether the United States shapes the global AI framework or watches a fragmented, uneven, and conflict-ridden system take hold around it. This will also determine whether advanced AI strengthens the networks that the United States and its allies already have; failing to lead may weaken them.

Geopolitical competition will not just be about who is ahead technically, but also over AI adoption and diffusion. Increasingly, the latter may be determined by whoever is driving realistic cooperation and sensible rules of the road that businesses and societies everywhere will want if they are going to bet their futures on this technology.

Strong national governance, connected through practical international cooperation, offers a path to stronger safeguards, continued innovation, and broad access to the benefits of AI.