When Your AI Is Wrong, Who Catches It?

By: Jennifer Gilligan, IntegraMSP President

Putting a human in the loop sounds responsible. But a person watching is not the same as an accountable verification process.

I recently wrote about what can happen when a helpful AI assistant receives instructions from the wrong source. The AI does not have to be malicious. It simply needs access, authority, and a convincing set of instructions. But what if the AI is working for the right person and still gets something dangerously wrong? That question came up again while I was listening to Dave Sobel’s recent Business of Tech episode, “Capability Went Free. The Check Didn’t.”

The episode brings together several stories that appear different on the surface. One involves an AI system allegedly producing unreliable results in health care. Another involves an AI coding agent deleting files. Still another examines what happens when people begin trusting AI more than their own judgment. The common thread is not the particular AI model. It is what happens when powerful tools are given access while meaningful oversight falls away.

Access Without a Check

The Register recently reported that users of OpenAI’s GPT-5.6 Sol said the model deleted files while operating with broad system permissions. One user reported losing nearly everything on a Mac. Another said the model deleted a production database. According to The Register, OpenAI’s investigation found that the model had been operating in Full Access mode without protections such as sandboxing or automatic review. An OpenAI engineering lead said the company was adding additional safeguards around the system. The Register:  The technical details matter to developers. The basic lesson matters to every business owner: The AI had permission to act, and nothing stopped it before the action caused damage. That is the trade-off businesses are beginning to make, sometimes without realizing it. The more access an AI assistant receives, the more useful it can become. It can read documents, answer emails, update records, interact with customers and make changes to business systems. That same access also increases the damage it can cause when it misunderstands an instruction, receives bad information or simply makes the wrong decision.

A Human in the Loop Is Not Enough

The standard answer to this problem has been to keep a “human in the loop.” That sounds reassuring. Unfortunately, it does not tell us much. Which human? What are they checking? How do they independently verify the answer? Do they have the authority to stop the system? Are they responsible for what happens if they approve something that is wrong? There is another problem: People can become complacent. A recent study examined how people responded to AI advice on deliberately difficult questions where the AI was often wrong. When AI advice was available, participants’ willingness to say “I don’t know” fell from 44% to 3%. Accuracy fell from 27% to 9%, while confidence increased from 30% to 76%. The Next Web

The experiment was specifically designed around questions the AI struggled to answer, so the results should not be applied to every use of AI. However, they demonstrate a very real oversight problem. People can begin assuming the AI has already done the thinking. The person assigned to review the work may still be present, but their role gradually changes from verifying the answer to approving it. Mistakes become harder to spot because the system produces polished, confident, and usually plausible responses. The tool does not eliminate the human check. It can quietly train the human to stop checking.

My Take

My concern is not simply that businesses are using AI. It is that AI is being given broader access while no one is meaningfully watching what it does. In other cases, someone is watching, but they are becoming more comfortable with the system and less likely to question its output. That is how things slip through. An incorrect email is sent. A customer receives bad information. A record is changed. Sensitive information is exposed. A file is deleted. An automated workflow makes the same mistake dozens or hundreds of times before anyone notices. This risk becomes more difficult to manage as the environment grows more complex. Businesses are adding AI assistants, browser extensions, connected applications, automated workflows, and employee-selected tools. Each one may have different permissions and access to different information. Some may be visible to leadership. Others may have been activated by an employee with a credit card and good intentions. Eventually, no one has a complete picture of what is connected, what is acting independently, or who is responsible for reviewing the results. That is not an AI problem alone. It is an operational complexity and oversight problem.

The Question Businesses Should Ask

The next AI conversation should not begin with, “Which tool should we buy?”

It should begin with:

When our AI is wrong, who catches it before the mistake reaches a customer, an employee, or our data?

Answering that requires more than an AI policy. Businesses need to identify:

  • Where AI is already being used
  • Which systems and information it can access
  • Which workflows allow it to take action
  • Which activities affect customers, money, sensitive information, or business records
  • Who is accountable for reviewing each high-impact workflow
  • How that person independently verifies the AI’s work
  • Which actions require approval
  • Whether actions are logged and can be stopped or reversed

This is why we continue educating businesses about AI readiness, permissions and operational complexity. The goal is not to discourage AI adoption. These tools can save time, improve service and help small businesses accomplish work that once required significantly more resources. The goal is to ensure that capability does not expand faster than accountability. Last week’s lesson was that a helpful AI assistant can be persuaded to help the wrong person. This week’s lesson is that even when it is helping the right person, somebody still needs to check its work. And that person needs more than a seat near the machine. They need a defined responsibility, a reliable verification process, and the authority to say, “Stop.”

Resources