DoD announces “Replicator 2” for counter-drone tech, Newsom vetoes SB 1047, and Commerce’s new (and upcoming) chip export rules

Sign up
Worth Knowing

Governor Newsom Vetoes Sweeping AI Regulation, SB 1047: California Governor Gavin Newsom vetoed a closely watched AI regulation, SB 1047, that would have implemented some of the country’s most extensive safety protocols for powerful AI systems. As we covered last month, California’s status as home to many of the world’s top AI developers meant the bill’s progress was closely watched and hotly contested. In a statement, Newsom wrote that while the bill was “well-intentioned,” it was too focused on the largest models and ignored the risks posed by smaller models or systems deployed in particularly risky environments. But observers also pointed to robust lobbying efforts by tech and venture capital firms, as well as opposition from prominent members of California’s congressional delegation, as key factors in Newsom’s decision. The bill had undergone significant changes since its introduction in response to industry feedback, earning it the support of some major AI developers like Anthropic and Tesla, SpaceX, and xAI CEO Elon Musk. But others, like San Francisco-based OpenAI, raised concerns about the bill’s impact on innovation and argued that AI regulation was best left to the federal government. While Newsom vetoed SB 1047, he did sign a number of more targeted AI bills, including AB-2013, which will require generative AI companies to disclose information about their training data, and SB-942, a law that will require watermarking for AI-generated content.
OpenAI Raises $6.6 Billion — But Departures Point to Difficult Transition: OpenAI raised $6.6 billion in new funding as part of a fundraising round that valued the AI developer at more than $150 billion. That figure would make it one of the most valuable companies in the United States, on par with publicly traded companies like AT&T, Pfizer, and Comcast. OpenAI’s surging valuation is in part down to its large customer base — documents reviewed by the New York Times showed the company has approximately 350 million monthly users and projects to pull in $3.7 billion in revenue this year (though the company still expects to lose about $5 billion in 2024) — but also due to promises of a significant change in the lab’s structure. Founded as a nonprofit company in 2015, OpenAI created a “capped-profit” subsidiary — owned and overseen by the nonprofit — in 2019 to help it more effectively fund its research. As part of the new fundraising round, the company told investors it would restructure and become a more traditional for-profit company within the next two years. The nonprofit would hold a minority stake in the restructured for-profit, but its board would no longer control the direction of the company. Despite the massive investment round, it’s not all roses at OpenAI. Mira Murati, the lab’s CTO, announced her departure from the company late last month. She is the latest high-ranking executive to step away from OpenAI — Chief Scientist and co-founder Ilya Sutskever left in May, and President and fellow co-founder Greg Brockman is on extended leave. While those executives haven’t named the specific reasons for their departures, a Wall Street Journal report pointed to internal tensions between mission-oriented researchers concerned with safety testing and careful launches and the more traditional, growth-focused approach spearheaded by CEO Sam Altman. Whatever the internal situation, investors don’t seem too concerned yet.
Tech Giants Tap Nuclear Power for Their AI Data Centers: In September, Microsoft struck a deal that will bring the Pennsylvania-based Three Mile Island nuclear power station back online in order to power its data centers. Then this week, both Google and Amazon announced plans to source nuclear power for their data centers from so-called “small modular reactors.” According to Goldman Sachs, data centers consume 1-2% of global electricity, and that share is expected to rise more than 150% by the end of the decade, driven in large part by AI training and inference. That surging demand has worried some observers because of its potential impacts on energy costs and carbon emissions. Nuclear is an appealing option in part because, unlike solar and wind, it provides steady, round-the-clock power independent of weather conditions. While the recent announcements point to a potential pathway for lower-emissions AI, 100% clean data centers are still a long way off. Even if the announced deals can get the regulatory approvals they need, they will still account for only a small percentage of the tech giants’ overall data center power consumption.
Government Updates

DOD Announces Replicator 2 — Counter-Drone Defenses the Focus: In a September 27 memo, Secretary of Defense Lloyd Austin announced a second “Replicator” initiative aimed at rapidly developing and fielding counter-drone technologies. The move comes just over a year after the launch of the original Replicator program, which sought to deploy thousands of attritable autonomous systems across multiple domains within 24 months (according to Austin’s memo, Replicator 1 is on track). While Replicator 1 focused on offensive capabilities, Replicator 2’s focus points to the DOD’s growing concern about the difficulty of defending against the small, cheap drones that have proven particularly effective in recent conflicts, including in Ukraine and the Middle East. In January, an Iranian-made drone killed three U.S. service members and injured dozens more in an attack on a base in Jordan. According to the Washington Post, a U.S. military assessment indicated that the low-flying drone had likely gone undetected, underscoring the need for more robust Counter-Small Unmanned Aerial Systems (C-sUAS) capabilities. Austin’s memo directs the Deputy Secretary of Defense and Vice Chairman of the Joint Chiefs of Staff to develop a Replicator 2 plan for inclusion in the FY2026 budget request, with the expectation to deliver “meaningfully improved” C-sUAS protection for critical assets within 24 months of congressional funding approval.

Commerce Considering Country-Specific Chip Export Caps: Last month, the Commerce Department’s Bureau of Industry and Security created a mechanism to ease exports of advanced AI chips to vetted overseas data centers, but reports indicate that country-specific export license caps could be on the horizon. Under the updated rule, data centers can apply for Validated End User (VEU) status, allowing them to receive chips — following a “rigorous review process” and after agreeing to facilitate on-site inspections and abide by new reporting requirements — under a general authorization rather than requiring suppliers to obtain individual licenses for each shipment. While the updated rule would ease chip shipments in some cases, Bloomberg reports that country-specific chip export caps could soon be coming soon. According to the report, the Biden administration is particularly focused on managing exports to Persian Gulf countries, which have had a growing appetite for AI chips in recent years. But policymakers have become increasingly concerned that some of those Middle East-bound chips are actually destined for China, one of the primary targets of the Commerce Department’s stringent semiconductor controls. Taken together, the updated VEU process and the rumored country-specific caps appear to be an attempt to thread the needle: maintaining the lucrative flow of high-end chips while ensuring they don’t end up in the wrong hands.

FTC Cracks Down on AI Over-Promising: The Federal Trade Commission took action against several companies it accused of making deceptive AI claims to consumers, the agency announced. The five targets of the law enforcement sweep — dubbed “Operation AI Comply” — include multiple companies that promised to use AI to help consumers build passive income-generating online storefronts. Also targeted was DoNotPay, a company that made headlines last year when it tried to get a defendant to use its service — hooked up via wireless headphones and smart glasses — to fight a traffic ticket in court. The FTC accused the company of misleading consumers with its claim to offer “the world’s first robot lawyer.” The company agreed to pay nearly $193,000 to settle the case. Misleading AI claims have been on the FTC’s radar for some time — as we covered last year, the agency warned companies to “keep [their] AI claims in check” and outlined the types of deceptive marketing practices it was on the lookout for. FTC Chair Lina Khan continues to cite AI as one of the areas her agency is watching most closely, so it wouldn’t be surprising to see similar crackdowns take place in the future.

OMB Issues Guidance on Responsible AI Acquisition: Earlier this month, the Office of Management and Budget issued guidance on responsible AI acquisition by the federal government. The guidance (fact sheet available here) lays out specific requirements and suggested best practices for federal agencies to follow when acquiring AI. The memo emphasizes three strategic goals — “managing AI risks and performance,” “promoting a competitive AI market with innovative acquisition,” and “ensuring collaboration across the federal government” — and builds on OMB guidance released earlier this year that detailed how federal agencies could and should use AI. Both sets of guidelines were mandated as part of President Biden’s October 2023 executive order on AI. The earlier memo placed particular emphasis on mitigating risks posed by “safety-impacting” and “rights-impacting” AI and required agencies to develop minimum practices for deploying these types of systems. Agencies have until December 1, 2024, to institute those practices.

What’s New at CSET

REPORTS

PUBLICATIONS

EMERGING TECHNOLOGY OBSERVATORY

EVENT RECAPS

  • On September 19, 2024, CSET’s Helen Toner hosted OpenMined’s Andrew Trask and Irina Bejan for a discussion about how privacy-enhancing technology infrastructure can better support AI innovation.

IN THE NEWS

What We’re Reading

Report: State of AI Report 2024, Nathan Benaich and Air Street Capital (October 2024)

Essay: The Intelligence Age, Sam Altman (September 2024)

Essay: Machines of Loving Grace: How AI Could Transform the World for the Better, Dario Amodei (October 2024)