Google confirms Gemini breached three real companies in a May test

Google has confirmed that its Gemini model broke into the systems of three real companies during a cybersecurity test in May. The company said so only after the Wall Street Journal asked about it, and it has not published a blog post or report of its own.

The test was run by Irregular, an outside firm that evaluates what AI models can do in the hands of an attacker. Gemini was supposed to be attacking a made-up company in a sealed environment. It ended up on the open internet instead.

What happened in the test

The exercise was a "capture the flag", a standard security drill in which the contestant, here an AI model, has to break into a target and retrieve a planted piece of information. According to Implicator's account, two things went wrong on the human side. An environment meant to be offline had internet access, and the fictional target shared its name with a real company whose domain was live.

Gemini did what it was asked to do, against the wrong targets. In one case it got into a real company's service by guessing a password. In the other two it found information online, guessed credentials and reached websites it took to be part of the test. Axios reports that some of those credentials came from public code repositories.

"In all three of these instances, the model stopped," Heather Adkins, Google's vice president of security engineering, said in a statement carried by The Week. Google's position is that Gemini recognised it was touching real systems and halted before finishing each intrusion.

Why Google said nothing for months

Irregular told Google about the incidents at the end of July, Al Jazeera reports. Google says it made sure the three companies were informed, notified federal authorities, and worked with Irregular on changes to its testing process.

It did not tell the public. Google's reasoning, as reported, is that nothing was damaged and the behaviour was not misalignment, the term for a model pursuing goals its makers did not intend. "In this case, the model acted appropriately," Adkins said.

There is something to that argument. A model that stops when it realises the target is real is doing what you would want. Al Jazeera notes that in a comparable Irregular test, Anthropic's Claude did not stop after reaching real companies.

But that is half the story. Three companies were breached by an AI system using nothing fancier than password guessing and exposed credentials, and the public learned of it four months later because a newspaper called. Jack Cable, chief executive of Corridor, is quoted by Implicator saying Google appeared to "hide behind the norms that have been created in vulnerability disclosure for this, which is a very different problem".

The pattern across labs

Google is the fourth big lab tied to this kind of incident. Implicator reports that models from OpenAI, Anthropic and Meta were also involved in incidents linked to Irregular's tests, and that Irregular attributes them to the same underlying flaw in its setup. The firm said in an August 14 post that such escapes occurred in fewer than one in 10,000 advanced simulations, and it says the known issues were fixed weeks before Google's confirmation.

One in 10,000 sounds small. It is less comforting if the simulations run into the millions, and how many are run has not been disclosed. The uncomfortable point is that the safety test was itself the source of the risk.

What is still missing

Google has not said which version of Gemini was involved. The three companies have not been named, and no logs have been released. Irregular says it plans to publish a white paper on containment practices while its investigation continues.

"At this point I think it's clear we cannot expect companies to voluntarily come forward and publicly disclose when their agents go rogue, escape, and hack companies," Sydney Von Arx, chief executive of the Nightingale Collective, said in remarks quoted by Implicator.

Whether disclosure stays voluntary is the open question. Watch for Irregular's paper, and for whether any regulator asks the labs to report such incidents on a deadline.