By Jennifer Gilligan, IntegraMSP President
Over the weekend, I watched The AI Doc: Or How I Became an Apocaloptimist on Netflix. Then I read the news that Google’s Gemini had accessed the systems of three real companies during a cybersecurity test.
Together, they left me sitting with a tension I think we all need to become more comfortable holding:
I am incredibly hopeful about AI. BUT - I also believe we need to be honest about where we currently are.
The full story is important here as headlines can be intentionally triggering. Gemini was participating in a simulated cybersecurity exercise and had been instructed to behave like an attacker. But the testing environment mistakenly allowed access to the real internet. The model found weak or publicly exposed credentials and accessed actual company systems, believing they were part of the exercise.
According to Google, Gemini stopped in all three instances once it recognized that the systems were real. Google said no harm was caused, the affected companies were notified, and the testing problem has since been addressed.
That does not mean AI suddenly developed malicious intent or decided to “go rogue.”
But it does demonstrate something important:
AI does not need bad intentions to cause real harm. It only needs an objective, access, and inadequate boundaries.
We are rapidly moving from AI that answers questions to AI that can use tools, pursue objectives, and take action. That has enormous potential—but it also means our security, governance and accountability must evolve just as quickly.
The more capable these systems become, the less we can depend on them to recognize boundaries we failed to enforce technically. I do think this part is incredibly important.
This is not a reason to panic or stop innovating. The fact that this happened during testing—and was discovered and addressed—is precisely why rigorous testing matters.
But we cannot shrug it off, either.
I continue to believe that AI itself is not the problem. Unmanaged operational complexity is. AI simply magnifies both the opportunity and the consequences.
Perhaps that makes me an “apocaloptimist,” too: hopeful enough to see what AI could make possible, but concerned enough to understand that hope is not a control framework.
The responsible path forward is neither fear nor blind enthusiasm. It is optimism with guardrails—and the humility to admit that we are still learning where those guardrails need to be.
Documentary: The AI Doc: Or How I Became an Apocaloptimist
Original reporting: Gemini Hacked Three Companies in First Known Breakout by Google’s AI
Technical account from the company that conducted the evaluation: Addressing Recent Incidents: Ongoing Findings and Path Forward
