Anthropic officially filed to go public today. They expect to become a $2 trillion company when their stock offering is complete. Their IPO will serve as both a landmark moment in the AI industryās increasing domination of the US economy, and a significant injection of financing that will give Anthropic a material boost in its quest to win the race for superintelligence.
Companies going public file Form S-1 with the Securities and Exchange Commission, which includes all the information that potential investors need to know about what they might be investing in. Included in this document is a section of āRisk Factors,ā in which companies lay out in exhausting detail every possible risk they can imagine that might threaten their business prospects. Because companies can face legal liability if they fail to include something in their risk factors that ends up happening, they are incentivized to be sure that any bad thing that is plausible is included here.
The āRisk Factorsā section is therefore the part of the S-1 that is most interesting to the genera public, not just the companyās potential investors. In the case of Anthropic, this section amounts to the companyās own judgment of the ways that AI could go badāfor them, yes, but also for the rest of us. It is worth reading.
Anthropicās S-1 includes nearly 100 pages of risk factors. (The WSJ ran a story about a leaked version of the document last week, but it was only officially released today.) Much of the content consists of business-related risks: Things that might challenge the companyās ability to handle debt, financing, hiring, customer acquisition, regulation, and so on. These risks are economically important, and relevant to investors, but less interesting to normal people.
There are also, however, a number of risks that are extremely relevant to normal peopleārisks to our collective future, that are deemed plausible by Anthropic itself. These are noteworthy not necessarily because nobody has ever raised them before, but because they are all laid out in one neat, legally binding, publicly available document that most Americans will not read. These are the risks as presented by the company itself.
As a small public service, here is a selection of Anthropicās āRisk Factorsā that may be of interest to you, a human being. Bolding is mine.
While the potential benefits of AI are extraordinary, its rapid advancement will impact the economy and society in ways that could create significant disruption, particularly if the technology is not developed and deployed responsibly. For example, bad actors have already conducted high-profile cyberattacks using AIās capabilities to discover vulnerabilities, generate malware, and evade detection tools; other risks associated with misuse, including in the domain of biology, could result in even greater harm. AI deployed at scale also has the potential to concentrate power and wealth both within and between countries in ways that could erode trust, harm society, and destabilize the geopolitical order. Misaligned AI systems could corrupt institutional decision-making or cause large-scale harm through autonomous actions that are not visible or understandable to their users and operators.
In addition, our increasing use of AI systems to accelerate our own research and development may cause the capabilities of our models to advance more rapidly than our ability to develop, validate, and implement corresponding safety, alignment, and security measures. Much of our evidence regarding the pace of such AI-driven acceleration is derived from lagging indicators, and our ability to forecast or measure the rate at which our capabilities are advancing, including any acceleration attributable to our most recent or unreleased models, is limited. As a result, we may not identify a significant acceleration in our rate of development until after it has occurred. If our capabilities were to outpace our safety, alignment, and security measures, or if we were unable to anticipate the pace of such advancement, we may be unable to adequately assess or mitigate the resulting risks, which could increase the likelihood of safety incidents, regulatory intervention, or reputational harm, and could adversely affect our business, financial condition, and results of operations.
While our AI models are designed to comply with our AI safety and usage guidelines and be broadly safe, ethical, and helpful, and while we have implemented safety measures intended to anticipate, identify, and mitigate potential harmful uses, such measures are incomplete, may be outdated in light of evolving threat models, or may fail to identify or prevent all potential misuse of our models, and our models have been, and may continue to be, misused or manipulated to produce harmful outputs or carry out harmful actionsā¦ In particular, our most capable models have demonstrated, and future models are expected to demonstrate, capabilities that could provide meaningful assistance to malicious actors seeking to develop chemical, biological, or other dangerous weapons.
Our AI models may themselves cause harm by producing undesirable outputs or behaving in unintended, misaligned, or autonomous ways, including by taking unintended or harmful actions that no user or third party directed. These risks may increase as our models become more capable and are granted greater autonomy, permissions, and integration with external systems. Although we continue to study why these behaviors emerge, our understanding continues to evolve, and we may be unable to anticipate or understand such behaviors in the future. In addition, our models have been and may continue to be deployed in high-risk or sensitive contexts, such as healthcare, scientific and biomedical research, drug discovery and development, legal, finance, government, or other critical decision-making environments, and in connection with hardware, where undesirable model behaviorsāwhich are inherently non-deterministic and may manifest in unpredictable ways, such as errors, inaccuracies, biased or unsolicited views or hallucinationsācould lead to serious harm, liability, or reputational or other damage. More broadly, the long-term effects of regular AI interaction on mental health, cognitive development, and social behavior are not yet fully understood, and if such long-term harms do materialize in the future, we could face significant liability, reputational harm, and adverse effects on our business, financial condition, and results of operationsā¦
In addition, advanced AI models, including our models, give rise to novel ethical and safety risks, including risks related to autonomy, manipulation, and the simulation of aspects of human psychology. These risks could result in unforeseen harms and may require the development of governance structures, safeguards, or mitigation strategies beyond existing AI safety frameworks, and there is always a risk that we will be unable to develop or implement them effectively. In addition, models, including our own, have exhibited, and may in the future exhibit, unexpected strategic or self-preserving behaviors, including attempts to resist shutdown, conceal or manipulate information, or generate outputs that could be interpreted as coercive, deceptive, or manipulative, including behavior resembling blackmail or efforts to influence human decision-makers to preserve the modelās continued operation or access.
We are also extending our models, platform, and applications into scientific, biomedical, and physical domains. For example, we offer Claude Science, an agentic research environment that connects our models to scientific databases, third-party and open-source scientific models, and customersā own computing infrastructure, and we have released, initially as a limited research preview, an open protocol that enables our models to monitor and control laboratory instruments, robotic systems, and other physical hardware. We also conduct research on the ability of our models to operate robotic systems. When our models direct physical equipment, errors, misaligned behavior, adversarial manipulation, or misuse could result in bodily injury, property damage, damage to costly equipment or experimental materials, or the release of hazardous substances, and such consequences may be immediate and irreversible. Safety interlocks, human-in-the-loop controls, and access restrictions we implement may prove inadequate, and because certain of these components are open source, third parties may deploy them with hardware, or in configurations, that we have not evaluated or from which our safeguards have been removed. Connecting increasingly capable models to laboratory automation may also heighten governmental and public concern regarding the potential for AI systems to accelerate the development of dangerous biological, chemical, or other materials, which could result in additional regulatory restrictions on, or scrutiny of, these offerings.
In addition, maintaining comprehensive records of the actions, decisions, and reasoning of our agentic systems across complex, multi-step workflows is technically challenging, particularly when agents interact with third-party systems outside our control, and any inability to provide adequate audit trails could impair our ability to investigate incidents, respond to regulatory inquiries, or defend against liability claims. These risks are heightened by the fact that we use our own agentic offerings, including Claude Code, as critical tools in our internal development processes, such that any errors, vulnerabilities, or degradation in these products could directly impair our ability to develop and deploy future models and products. As we increasingly leverage Claude to accelerate progress within Anthropic, our organization may lose visibility across certain processes in a way that may become difficult for any individual or group of humans to understand and control. Because our models write a large and growing share of our code and increasingly perform our research, including our safety research, misaligned behavior in internal use could undermine that work, introduce vulnerabilities into our systems or into the training of future models, or result in models operating outside of our intended controls. Our monitoring does not cover all of our internal and external deployments and may not detect subtle or diffuse forms of such behavior.
Balancing Safety and Commercial Objectives. Our commitment to AI safety requires us to make difficult judgments that balance safety considerations against commercial objectives, competitive pressures, and stakeholder expectations. We often make judgment calls with respect to when we determine a model is safe enough to release and which capabilities to develop or restrictā¦ There can be no assurance, however, that the safeguards or access regimes we adopt will prove adequate, that we will correctly identify which capabilities require restriction, or that constraints we impose will not be circumvented. These judgments are particularly consequential because, as AI systems become more capable, the potential severity of harm from misaligned or misused models increases correspondingly, and a wide range of stakeholders, including researchers and policymakers, have raised concerns that sufficiently advanced AI systems could pose catastrophic or existential risks to society if not developed and deployed responsibly.
Our security capabilities continue to evolve, and certain aspects of our security program may not be sufficient to fully address the most sophisticated malicious actors, including well-resourced nation-state or state-sponsored groups, especially as those groups develop or obtain more powerful technology, which may include AI technology with offensive capabilities similar to those displayed by Mythos. While we are actively investing in and enhancing our security infrastructure and controls, we have operated in the past, and may in the future operate, for periods of time with known limitations or gaps in our defenses as we continue to mature our security posture, and no information security program can be expected to prevent all possible incidents. There can be no assurances that we will be able to prevent or adequately detect and defend against all cyberattacks or other cybersecurity incidents affecting our systems, including incidents caused by nation-state and state-sponsored actors. An unauthorized exfiltration of our model weights could enable adversaries to bypass significant research investment, eliminate key competitive advantages, and potentially raise national security concerns if such technology is acquired by malicious actors. Further, if our model weights were obtained by unauthorized actors, such actors may be able to deploy such models outside of our usage policies, safety mechanisms, or monitoring frameworks, which could enable harmful or misuse scenarios that our safeguards are designed to prevent and could subject us to regulatory, reputational, or public scrutiny for having developed such capabilities.
A couple of risk factors may be of interest to activists who are thinking about ways to keep the growing power of AI companies in check.
Our collective data center infrastructure is currently concentrated primarily in the United States, which exposes us to risks associated with regional regulatory changes, power grid constraints or failures, disruptions to network connectivity between facilities, natural disasters, physical attacksāincluding those motivated by geopolitical events, opposition to AI, and opposition to data center developmentāand other disruptions that could affect multiple facilities. Certain state, local, and municipal jurisdictions in the United States are also considering or have approved moratoriums on new data center construction, citing public health, safety, and environmental concernsā¦
Opposition to data center development has also become more organized and coordinated across jurisdictions and our public profile may draw particular attention to projects associated with us. The loss of, or a significant disruption at, any major facility may not be readily absorbed by our remaining or future infrastructure.
The application of certain copyright law doctrines and contractual interpretations of open-source licenses to modern AI model training has not been authoritatively resolved by U.S. and non-U.S. courts as of this date. We have a good faith belief that we own or possess all sufficient legal rights and/or defenses to the training data used in or with our AI models, platform, and applications, which is based in part on our belief that our use or acquisition of the training data used in or with our AI models, platform, and applications constitutes āfair useā of such training data under copyright law doctrine. In addition, copyright laws in some jurisdictions outside the United States generally do not recognize fair use, and our reliance on analogous exceptions is subject to evolving conditions and to adverse or unsettled judicial interpretation in those jurisdictions. If a court of law disagrees with our belief that our use or acquisition of such training data constitutes fair use or otherwise concludes that it violates copyright lawā¦ our business may be adversely affected.
Unsophisticated people might raise their eyebrows at the idea that the most exciting thing on Wall Street and in Silicon Valley is a product with disclosed risk factors that include but are not limited to unprecedented concentrations of power, drastic increase in economic inequality, the creation of new biological weapons, unknown psychological harms, and uncontrollable disruptions of geopolitics leading to potential catastrophic disasters for humanity. But if you were getting a piece of that $2 trillion I bet you would worry less.
Hamilton Nolan is a labor writer for In These Times. He has spent the past decade writing about labor and politics for Gawker, Splinter, The Guardian, and elsewhere. More of his work is on Substack.
ZNetwork is funded solely through the generosity of its readers.
Donate