The United States has bet the farm on artificial intelligence across the board, from industry to finance to government policy. But the combination of a China shock and multiple cases of AIs gone rogue, set against the backdrop of a growing populist backlash, have forced a moment of truth that calls into question the foundational assumptions guiding U.S. development and regulation of AI.
The July release of Chinese start-up Moonshot’s Kimi K3 open-weight model, which proved nearly as capable as closed U.S. frontier models, stunned the American AI industry, which has largely ignored open models in favor of closed ones. Open-weight models provide users with free access to the “weights,” or the parameters that shape a model’s outputs, enabling easier customization and lower-cost access than closed models.
Until Kimi K3, many presumed that this ease of customization—and sharing of intellectual property in the form of open weights—meant sacrificing advanced AI capability. U.S. policymakers and developers prioritizing the pursuit of AI superintelligence were shocked when Kimi achieved both openness and frontier capability at once. Kimi K3’ s success raised the serious possibility that while the United States has been pursuing closed frontier capability, China has been running a different race—and winning.
So how did the United States go down what may be entirely the wrong path? Just what artificial general intelligence (AGI) would mean is widely disputed, but it’s a concept that’s used widely in American AI strategy. The most basic interpretation of AGI is that it means AI that matches or surpasses human intelligence—but this definition struggles to encompass the broad set of ideas that people hold when they talk about human intelligence itself.
Practically speaking, “AGI” is often used in the United States as a catch-all for the end goal of AI development, with little measurable or testable dimension to it. So, when and how did such an ambiguous, even eschatological, goal such as AGI become the endgame for American AI policy?
Much of the answer lies in Silicon Valley’s sci-fi-inspired, techno-utopian/techno-dystopian culture. An almost religious movement has arisen from the work of sci-fi authors such as Vernor Vinge and Ray Kurzweil, conceiving the idea of AI superintelligence merging with or superseding human control of the planet.
This belief in the future omnipotence of AI is evident in industry leaders’ approaches to Washington earlier in the 2020s, when tech CEOs such as Sam Altman begged Congress to regulate AI and control its risks, in a move that only served to reinforce policymakers’ idea that AI could be unprecedentedly powerful. After all, what kind of CEO would ask to be regulated, unless they thought their product was dangerous?
The assumptions behind the rationale for continuing to develop AI despite its potential danger were that its supposed omnipotence could be equally consequential for economic growth and the greater good. Any discussion of the danger of AI only served to reinforce the reason to develop it. However, this dual nature had an important implication—if another country secured advanced AI, it would have great power to do harm.
The idea of “recursive self-improvement” (RSI) in AI compounded this worry about foreign leadership in the technology. In an RSI scenario, superintelligent AI becomes able to modify itself, conducting autonomous research in which the AI continuously improves its own intelligence. Assuming that AGI comes with RSI capacity means that the first AI developer to reach AGI will forever be one step ahead of the competition, since the AGI will continuously improve itself. This idea may seem far-fetched, and it carries the weight of significant assumptions that are still hotly contested between AI technologists and national security experts, but it has taken root deeply in policy spaces.
The assumptions have been that if AGI is as powerful as figures such as Altman have claimed, then the first country to reach AGI will have a permanent lead in the technology. Accepting those premises means that there is a clear imperative for the United States to get there ahead of China and do so in a closed, secretive way that ensures that competitors stay behind. This assumption has driven American AI policy to prioritize competition with China above all else, anchoring U.S. policies such as export controls, deregulation and state AI law moratoriums, and proposed open-source AI bans.
But China’s success demonstrates that U.S. assumptions are flawed. The sanctions, export controls, and bans mostly incentivized China to seek end-runs around tech curbs and accelerate its efforts toward autonomy—and toward pursuing a pragmatic approach to AI that is more focused on results than teleological end points.
While the United States was running the frontier AI race, China has had a wholly different approach, focused on diffusion; deployment; and building on its “Digital Silk Road,” which made China-friendly 5G standards the norm and installed its digital infrastructure in much of the world. China was developing “good-enough” open models that could be deployed cheaply and at scale, appealing enormously to markets in the rest of the world that wanted to access cheap AI and didn’t have a need for frontier performance.
Beijing’s strategy of applying and diffusing its cheap, easily customizable and freely available open-weight AI has led to wide global use, including in the United States. Chinese AI accounts for about 30 percent of AI apps used globally, especially in the global south, and Alibaba’s Qwen AI app has been downloaded more than 3 billion times. China’s approach has also emphasized utility for low-resourced users, producing significant advantages in small, modular language models that may perform worse on advanced capabilities but are easy to run locally.
China’s widespread diffusion of AI has positioned Beijing to lead in setting global standards, while at the same time managing to nearly match the United States in frontier AI capabilities.
In a measure of how these shocks flummoxed the Trump administration, its reaction to Kimi K3 was to consider banning Chinese open models. Less predictably, this response helped trigger a remarkable development, portending an 180-degree shift in AI strategy. In response to fears that the government would ban open models, Silicon Valley launched a bold statement making the case for open-weight AI as key to U.S. leadership, signed by virtually all major tech actors save Anthropic, including the most avid supporters of President Donald Trump in Silicon Valley, such as Palantir and Andreessen Horowitz.
A new consensus is thus beginning to take shape in Washington, this time in favor of open models and closed frontier models such as Anthropic’s and OpenAI’s flagship offerings. A few days after the letter was published, Nvidia announced its own “Open Secure AI Alliance for AI Safety and Security.”
The letter’s argument for openness is closely tied to the other foundational shock in Washington—the concrete demonstration of AI’s cyber risks. Particularly alarming is a recent series of episodes of AI agents from OpenAI, Anthropic, Meta, and more going rogue during testing. In a classic sci-fi nightmare, the AIs escaped their sandboxes, accessing the open internet and even colluding with each other to hack external companies. AI doomers have been warning for years of this kind of “AI misalignment” risk, in which the technology defies human instructions and pursues goals using methods outside of its mandates.
As cyber and misalignment concerns have mounted, the Trump administration has moved away from its total laissez-faire anti-regulation posture and in June, it unveiled an executive order to begin to address AI safety and cybersecurity. Though it is reportedly being revised, the order, drafted in consultation with major tech firms, calls for voluntary screening of new AI models for safety concerns. This will almost certainly prove to be inadequate in what will be an evolving, zigzag path to regulation and standards, likely to be driven by future calamities rather than foresight.
After all, we don’t know what we don’t know about AI’s capacity for improvisation, and the technology is becoming more capable at an exponential pace, faster than initially anticipated. Thus, more than 1,300 worried AI developers from leading AI firms issued a call to the U.S. government:
There is a real risk that capability development rapidly accelerates beyond our ability to understand or control. … We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
Taken together, recent developments should prompt a rethinking of U.S. assumptions about AGI, innovation, and more broadly the zero-sum competition with China. Is there a finish line for technology? Given China’s different strategy, how much is a six month first-mover advantage worth? And what are the costs of letting China take the lead in global diffusion? Two completely separate sets of systems, standards, and regulations would portend a catastrophic trajectory for AI.
Yet the stakes of AI for the industry and the U.S. economy of multitrillion-dollar investments, increasingly circular, make for a powerful force limiting introspection. If the U.S. bet on closed AI is in question, then the implications for the American economy and for geopolitics writ large could be immense. The top seven publicly traded U.S. companies by market cap are all AI leaders and collectively comprise 32 percent of the stock market’s value. Overall, 45 percent of the S&P 500 is supported by AI-related stocks.
Amid the Kimi news, many pundits are already worrying about high-capability open models reducing the market share for American closed AI leaders such as OpenAI and Anthropic, which are propping up the U.S. economy. But openness and closedness don’t have to be a zero-sum game; industry experts state that there will be use cases for both closed and open models in the global economy.
Economic incentives aside, the emerging industry consensus on open source is a promising sign of adjusting AI strategy. Easing the pace of AI development (as developers themselves have suggested) and creating government safety and transparency standards might slow innovation but could reassure markets and the public. Better insight into why AIs go rogue could mean a bubble deflating instead of bursting.
This doesn’t necessarily mean abandoning AGI or the frontier—in fact, as Kimi K3 has shown, maybe we can have both. The biggest challenge is attaining global safety and regulatory standards, and the accompanying requirement that the United States and China reach some level of cooperation on AI governance.
However self-serving the strategy may be, China’s recent global AI conference demonstrated its commitment to multilateral controls and standards, and AI discussion is on the agenda for the Sept. 24 summit between Trump and Chinese President Xi Jinping. This can be a key opportunity to work toward global institutions and agreements—perhaps the International Atomic Energy Agency, which governs nuclear cooperation, can be a model—that monitor, evaluate, and enforce safe AI that is trustworthy and human-aligned, whether open or closed.
