Before You Paste Secrets into an AI Tool, Look at the Tuition Samsung Already Paid

A while back, I came across a news story that sent a chill down my spine.

Last year, the UK High Court heard a major case with £89 million in damages at stake. The legal brief the lawyers submitted cited 45 precedents. When the court checked, 18 of them didn't exist.

The AI had made them up.

Can you believe it? A filing from a team of professional lawyers — nearly half the precedents in it were fabricated out of thin air. The judge sent out an urgent letter warning the entire legal profession: don't put too much faith in this stuff.

That story made me want to talk seriously about one question: how exactly does generative AI hurt a company, step by step?

What Is Generative AI, Exactly?

If you use ChatGPT every day to draft copy, or Copilot to fill in code, you may never have stopped to ask this question.

Put simply, generative AI is the kind of thing where you give it one sentence and it hands you back a finished product. Text, images, code, voice — it can generate all of them. ChatGPT, Gemini, Copilot, Grok — they're all in this family.

The efficiency it brings is real.

In early 2025, Bain & Company ran a survey: 95% of U.S. companies were already using some form of generative AI. In just over a year, the use cases that had actually made it into production doubled. More than 80% of companies said the results met or even exceeded expectations.

McKinsey has run the numbers too: generative AI could add $2.6 trillion to $4.4 trillion a year to the global economy. Marketing, customer service, coding, and R&D carry the biggest weight.

Good grief. $4.4 trillion — bigger than the GDP of the vast majority of countries.

But.

Guardrails? Only 27% Have Them

Everyone can see the upside. But the guardrails?

There's another number from that same McKinsey research, and it left me silent for a long time: only 27% of companies run a full human review of AI output before it gets delivered and used.

That means nearly three out of four companies let AI output flow straight to customers, to the market, to the courtroom.

Add one more number, from a survey by the security vendor BlackFog: only 53% of employees can explain how the data they feed into AI gets stored, analyzed, or used.

More than half of employees are handing things to AI with their eyes closed.

In one line: gains are counted in seconds; risks are settled in years.

Let Me Tell You Four Stories

"Risk" is too abstract a word. Don't worry — let me tell you four things that actually happened, and you'll get it.

The first story you've already heard: the UK lawyers' case from the opening. An £89 million case, and 18 fabricated precedents.

The second story stars Air Canada.

The chatbot on the airline's official website cheerfully told a passenger that bereavement fares existed, and that a ticket bought at that fare could be refunded. The passenger believed it and booked. When they applied for the refund, the official policy said: no such thing. They refused to let it go and took the matter to a tribunal.

How did the tribunal rule? The airline pays. The reasoning was blunt: what your AI says is what your company says.

The third story stars Samsung.

In 2023, some Samsung employees pasted meeting minutes and product source code straight into the public version of ChatGPT, hoping it would polish and debug them.

The trouble is, the public platform may use the data users type in to train its models. In other words, Samsung's code could show up in the answer to someone else's next question.

Once data is handed over, it never comes back.

Samsung's response was to ban the tool company-wide, then redesign how the company itself uses AI.

The fourth story is a little lighter: McDonald's.

In 2024, McDonald's piloted AI ordering at its drive-thrus. The result? Greatest hits like bacon landing on ice cream, and one order accidentally ballooning into a giant batch — the videos went viral. In the end, the pilot was scrapped.

Notice: the first two stories hit the wallet, the third hit the lifeblood, the fourth hit the reputation.

The bill for AI mistakes never comes in small amounts.

Five Pits, Each Deeper Than the Last

Take those four stories apart, and underneath them are really five pits.

Pit number one: data leaks.

Customer privacy, trade secrets, patents — once pasted into a public platform, consider them gone for good. Not only does it violate company policy; one step can land you on the wrong side of compliance red lines like GDPR and HIPAA. By the time you want to pull it back, you can't.

Pit number two: the attackers are using AI too.

You use AI to work faster; scammers use AI to scam faster. Writing a decent phishing email used to take some real way with words. Now a total novice can mass-produce scams indistinguishable from the real thing — and have AI write malware in batches and launch attacks on autopilot.

AI has dropped the barrier to crime all the way to the floor.

Pit number three: making things up with a perfectly straight face.

AI speaks fluently and confidently, but it hallucinates — it invents facts that don't exist. When an employee believes it, bad information seeps into the business process. When a customer believes it, you get the Air Canada bill. And if the training data itself carries bias, the output keeps amplifying that bias.

Pit number four: copyright and compliance.

AI is trained on oceans of public data. Content you generate with AI may unknowingly collide with copyrighted works. Publish it, and the liability for infringement is yours — not the AI's. Heavily regulated industries like finance and healthcare carry extra shackles: disclosure obligations and data-handling duties.

Pit number five: the collapse of trust.

One AI incident — say, publishing wrong information in public, or leaking a secret — damages a brand exponentially. In the social media era, bad news spreads across the entire web in a few hours. Customers, employees, investors, and regulators will all take a fresh, hard look at your company.

Rebuilding trust usually costs an order of magnitude more than prevention would have.

So What Do You Do? Five Things

That all sounds scary, but the guardrails aren't complicated. Darren Williams, BlackFog's founder, made a point I completely agree with: innovation and security have never been either/or. Put real controls in place, and the team can use AI with confidence.

How do you put them in place? Five things.

First, control where the data goes. Use only enterprise-grade AI tools that spell out clearly how data is stored. For sensitive information, stay away from the public free versions. Samsung has already paid that tuition on everyone's behalf.

Second, train your people properly. Most leaks start with human carelessness. Which data must never be pasted into AI, which platforms are officially approved — write it into policy, and repeat it again and again.

Third, watch your AI vendor contracts closely. Who owns the data, how long it's kept, who is liable when things go wrong, how security standards align — write every clause out clearly. Never assume a third-party tool is compliant by default.

Fourth, choose privacy-first platforms. If zero retention is possible, keep no data. If a zero-trust architecture is available, adopt it. The more heavily regulated your industry and the more valuable your intellectual property, the more you should sweat this one.

Fifth, make AI usage visible. Who is using it, where, and how — that must be traceable. The team should feel safe discussing topics like bias and misuse out in the open. Accountability has to be built into the process, not just a poster on the wall.

What Comes Next?

Three things are worth keeping in mind.

Regulation will only get stricter. Data privacy, model liability, boundaries on use — governments everywhere are tightening up. Don't wait for the rules to land before you catch up on the homework.

AI-powered attacks will only get more convincing. Phishing emails that imitate your writing style, synthetic voices indistinguishable from real ones — these are already in the scammers' hands. Some people even worry that one day there will be ransomware that can rewrite its own code and slip past detection.

And there's a softer but deadly problem: models keep getting more complex and harder to explain. Once AI's decision-making becomes a black box, what will you have left to tell your customers?

One Last Thought

Back to those 18 fabricated precedents from the opening.

It's not that the lawyer didn't know AI makes mistakes. He just never imagined it could be wrong with such confidence.

Generative AI is a money printer — that part is true. But next to the money printer, there must always stand a guard you posted yourself.

The tool doesn't take the blame; the company pays the bill. May AI print money for you — not pay tuition on your behalf.