AI Safety Needs Engineering, Not Apocalyptic Certainty

When AI researcher Jacob Coxon resigned from Anthropic, he warned that frontier laboratories were racing toward increasingly capable systems without adequate control over the risks. His concerns deserve attention. But the debate too often presents a false choice: accept predictions of catastrophe or be labeled an opponent of AI safety.

There is a more practical path.

Anthropic CEO Dario Amodei argues that AI capabilities may be advancing faster than safeguards. He proposes independent evaluation, democratic-country coordination and possible limits on systems that accelerate AI research.

Amodei identifies a legitimate problem. Frontier laboratories face intense competitive pressure. Each fears that slowing down will surrender leadership to another company—or to China. We cannot assume market incentives will produce adequate safeguards.

But estimates assigning a specific probability to human extinction are not scientific measurements. They are judgments about an unprecedented future. Presenting them with numerical precision creates a certainty the evidence does not support and distracts from risks we can address now.

Nvidia CEO Jensen Huang offers an engineering-focused response. Frontier laboratories possess information no outside organization can reproduce easily: access to their models, training systems, evaluations, incident records and technical teams.

Those laboratories should therefore have primary responsibility for building effective controls. After a material incident, they should determine what happened, why protections failed and how recurrence will be prevented. Solutions should be incorporated into secure operating environments, continuous monitoring, regression testing and enforceable release gates.

This does not mean companies should regulate themselves behind closed doors. Outside experts should challenge and verify their controls, while government ensures that serious findings cannot be concealed or ignored.

Elon Musk adds another practical proposal: frontier companies should test one another’s models before release.

Every major laboratory has developed a safety “test harness”—evaluations designed to identify dangerous cyber capabilities, weapons-related assistance, deception, safeguard evasion and other problematic behavior. But laboratories apply these tests to their own models. That is the equivalent of grading their own homework.

Under Musk’s proposal, a developer would provide several qualified laboratories with controlled access to a model before release. Each would apply its own tests. Because the organizations use different models, methods and assumptions, they are more likely to discover weaknesses the developer missed.

Companies would not need to exchange model weights, training data or trade secrets. Testing could occur through controlled access with actions logged. The developer would receive the findings, correct material problems and submit the model for retesting.

If a company ignored a credible warning and released the model anyway, evaluators could disclose the unresolved danger. That would create reputational consequences and evidence of negligence if harm followed.

We believe the strongest framework combines these perspectives.

Frontier laboratories should build the controls because they possess the data and expertise. Competing laboratories should challenge those controls with different testing systems. Independent evaluators should review disputed findings and prevent peer testing from becoming superficial industry self-certification. Government should establish minimum requirements, require disclosure of serious incidents and intervene when a company cannot demonstrate control.

The strictest obligations should focus on organizations capable of creating frontier-level risks—not every startup, university or open-source developer. Poorly designed regulation could protect dominant companies while weakening the independent research and competition needed to identify their mistakes.

AI safety is too important for either complacency or sensationalism. We do not need to prove catastrophe is imminent before demanding safeguards. Nor should speculative predictions become a substitute for evidence.

The immediate objective is clear: require the companies creating frontier risks to build strong internal controls, expose them to external challenge and demonstrate that serious failures have been corrected before releasing more powerful systems.

Author note

Remick prepared this commentary for Maine Vision 2040. It is adapted from a longer discussion paper examining AI safety, institutional influence and the effects of regulation on competition and open innovation.