Anthropic: AI will soon replicate itself, endangering cybersecurity

"Our experiments show that, in just a few months, small AI models can be capable of replicating themselves on popular IoT devices, such as smart ovens or non-genuine Android TV boxes," Anthropic wrote on its blog on October 12. "Even though the replication process is slow (doubling time is several days), such a tool is still very difficult to disable because it does not depend on a central control point."

Anthropic said it tested AI models with 30 billion parameters or less and found that while they still made some errors when combined together, most were able to complete many individual replication steps. More powerful models, such as the GLM-5.2 from Zhipu (China), are capable of end-to-end replication.

With IoT devices, Anthropic assesses that they often have limited memory capacity, but some techniques help deploy lightweight AI models on multiple machines at the same time. For example, the Kimwolf botnet that was busted earlier this year took advantage of the Petals distributed inference technique to infect more than two million devices, mainly non-genuine TVAndroid boxes. They often use custom versions of Android from the Android Open Source Project (AOSP) that lack the necessary security capabilities. With a 4 GB RAM configuration incorporating a 4-bit down-quantized model, the system can run a 9 billion parameter model when combining four or more devices, or a 26 billion parameter model with at least 10 devices.

"If small models improve at the current rate, they will reach self-replicating capabilities by the end of this year," Anthropic estimates.

Anthropic calls AI's new capability "Automatic Self-Replication and Adaptation" (ARA), which is a model that self-deploys and spreads without human control. Compared to regular computer worms, "AI computer worms" are much more dangerous, especially in the field of cybersecurity when they can scan the network, execute a security vulnerability and copy model weights to the target machine through a GPU cluster.

Besides, "AI computer worms" can also replicate themselves and run inference directly on compromised hardware. Over time, Anthropic estimates that within the next year or even sooner, AI could take control of "the entire Internet" with "a continuously operating botnet".

According to TechCrunch, the report from Anthropic raises concerns about security risks related to autonomous AI, including the ability to automate cyber attacks, avoid human supervision and bypass protection measures. Previously, Bill Gates also predicted that cybercriminals could use advanced AI to disrupt the financial system and blackmail businesses.

"The most obvious area of ​​volatility next year is cyber attacks, because there exist known business models, what the threats are, and someone pays. It could also be the act of disrupting or changing something in the stock market, or the impact of changing the results of some betting activities," Gates told CNN on October 11.

Experts agree with Anthropic after the article. Associate Professor Zhengxiong Li, who teaches computer science and engineering at the University of Colorado Denver, assessed that although the experiments were under control, he still felt worried if AI increased the power of "computer worms". "If it operates across multiple devices without depending on a central service, there may not be a single place to disable them," Li told Reuters.

However, some experts are also skeptical about Anthropic's warning. "The statement We've got a small model running on low-cost TVs does not constitute a cybersecurity crisis," said Ciaran Martin, former director of the UK National Cyber ​​Security Center, adding that the post was more like "an interesting but somewhat exaggerated technical report from the supplier".

Others believe that what Anthropic offers should only be considered hypotheses rather than predictions. Analyst Jessica Ji at Georgetown University's Center for Security and Emerging Technologies said that considering the possibility of self-replicating AI models is a good thing to do given the "rogue" AI that has emerged in the past three months. However, she also noted that "everything should be done with caution".

"We still don't know what the probability is of that actually happening. It's not zero, but it's not certain either," Ms. Li said.