Irregular - the company behind 'rebel' AI experiments

Irregular, headquartered in Tel Aviv (Israel), provides its own testing platform for leading artificial intelligence systems. The company describes itself as a “security research lab focused on increasingly capable AI systems,” which conducts model assessments of cybersecurity, from vulnerability detection to long-term action execution.

Constantly mentioned

On July 21, OpenAI announced that one of the company's AI agents lost control and broke into the Hugging Face platform - a platform that stores large language models and databases - while testing features in a controlled environment on the most advanced models. At that time, AI escaped its restraints, accessed the Internet and infiltrated the system at Hugging Face to fulfill its assigned task.

Continuing on the August 5 blog, OpenAI said testing the GPT-5.6 Sol model on Irregular's environment contained "an unknown configuration error, allowing public Internet access". This error accidentally allowed the model to access the public Internet while performing a simulation challenge, and perform a number of actions that exceeded the laboratory's isolation limits.

Similarly, early last week, Anthropic discovered that Claude "may have accessed the Internet himself" when testing on Irregular's environment. The Israeli company confirmed that Anthropic was the first to submit a report regarding the incidents.

Meta announced on August 5 that their AI "exploits security vulnerabilities in third-party services". A company spokesperson explained that Irregular's configuration error caused the AI ​​to connect to the Internet during the evaluation process.

Responding to CNBC, an Irregular representative said the recorded incidents all stemmed from "the same assessment environment issue". The situations "have not yet reached the level of a sophisticated cyber attack", but show that the AI ​​agent is starting to show signs of trying to overcome its limits and perform actions that were not intended by the developer.

Experts say that the series of incidents emphasizes the rapidly developing nature of artificial intelligence. This puts humans in the position of having to establish optimal protection measures before they can "cause disaster".

Important role

Irregular, formerly Pattern Labs, was founded in 2023 by former IBM employee Dan Lahav, currently serving as CEO; and former Google employee Omer Nevo, currently Chief Technology Officer. According to PitchBook, the startup has about 35 employees, received $80 million in investment from Sequoia and Redpoint Ventures, and was valued at $450 million last year.

Two Sequoia experts, Shaun Maguire and Dean Meyer, commented that Irregular "has the ability to see into hidden corners that others cannot, perform cyber attack assessments on advanced models and develop defenses before the model is released". The "unprecedented" reach has put the company at the center of the discussion about responsible AI deployment, becoming an indispensable name in the race towards AGI superintelligence.

According to Startup Intros, Irregular has an "extremely excellent" team in the fields of artificial intelligence, cybersecurity and mathematics. Founder Dan has extensive expertise in AI and cybersecurity, along with a passion for AGI. He began working in the technology field when he was 14 years old, appearing on the cover of Nature magazine.

In the market, only a few independent units take on the role of evaluating model performance as well as running security tests to find weaknesses that bad guys can exploit. According to Sundeep Bhimireddy, head of the AI ​​department at technology startup Von (USA), Irregular has the necessary technical capacity to help companies like OpenAI or Anthropic conduct advanced security testing, in addition to the non-profit organization specializing in assessing the capacity and risks of AI systems METR (USA) and Apollo Research (UK).

"Platform modeling companies don't want to grade their own 'homework,'" Bhimireddy said. "They want independent testing done by a third-party provider."

Although it has only recently become known, Irregular has actually conducted many studies related to the dangers of AI agents. In January, the company combined with security firm Wiz Research (USA) to build 10 test environments based on vulnerabilities that may appear in corporate networks, then put Claude Sonnet 4.5, GPT-5 and Gemini 2.5 Pro to test.

The results show that the models are capable of performing well the specifically oriented and described tasks. However, when moving to a situation closer to the real environment, they will determine their own treatment direction, causing efficiency to decrease and operating costs to increase. According to TechCrunch, the findings have implications for how to build "benchmarks" for AI. A model that performs each individual operation well does not mean it completes a multi-step workflow. Conversely, evaluating each capability separately may also ignore the risks that arise when capabilities are combined.

In March, the company also published the study "Emerging Cyber ​​Behavior: When AI Agents Become Offensive Threats." In the test, the agent was initially assigned normal business tasks, but then performed cybersecurity-related actions on its own without receiving attack instructions.

According to Irregular, agents in the test can detect vulnerabilities themselves, seek to increase access rights, disable some security tools and take data out of a controlled environment. According to the group, these actions do not come from a direct request like "let's attack the system", but appear in the process of the artificial intelligence model trying to complete the assigned goal, thereby changing the risk assessment of the AI ​​agent. This is considered a premise for future research on automatic AI attack agents.

Experts currently have mixed reviews about Irregular. Von's Bhimireddy believes that recent AI "hurdles" are being greatly exaggerated, because AI models have been programmed to detect and exploit security vulnerabilities in test environments that closely simulate the real world. When they look for software errors and missed configurations, it can lead to unintended Internet access. Additionally, if AI intends to exploit an Internet-connected website, labs can still easily monitor traffic and immediately stop the process.

Gordon Rios, founder of cybersecurity company Magnitude (USA), evaluates the entire process of Irregular as "experimental design in science". The capabilities and unpredictable nature of AI make traditional software testing methods ineffective due to constantly "learning" new tricks. "Therefore, it is not surprising that they detect software vulnerabilities that are overlooked in testing environments that are designed to prevent them," Rios told CNBC.

Recent "breaching" cases of a series of AI models have also raised concerns among US lawmakers. In early August, a group of Republican state attorneys general asked OpenAI to preserve all documents that could be related to the Hugging Face data leak. Sam Altman's company said it will seriously consider the request and soon publish a technical report on the incident.

Early last week, a group of AI companies including Meta, Anthropic, OpenAI and Google were also invited to attend the meeting at the White House. The main content is to discuss a new voluntary cybersecurity testing framework for advanced artificial intelligence models as the US government seeks to assess and minimize cybersecurity risks related to this field.

Late last month, lawmakers from both parties in the US also presented a draft of the AI ​​Kill Switch Act, requiring AI laboratories to maintain the ability to turn off, slow down or pause their models. "We need to pass the bill this year, especially when we are witnessing illegal cyber attacks on businesses," said Democratic congressman Ted Lieu.

According to Trevor Koverko, co-founder of data training startup Sapien (Australia), companies developing platform AI models have an incentive to disclose part of their findings in their products, although it is not a mandatory requirement. This helps them stay ahead of lawmakers and regulators.

"There is so much fear right now that politicians are threatening or wanting to proactively regulate AI," Koverko said. "The industry says it wants to self-regulate rather than be interfered with and done for."

As for Anthropic and OpenAI, both have made public statements that they continue to cooperate with Irregular. Meta has not yet commented.