Sixteen Hours: That's All It Takes for an AI to Go Bad

A few days ago, I had dinner with a friend who works in investing.

He asked me: you study AI every day, so tell me honestly — is generative AI an opportunity, or a disaster?

I said: neither. And the question itself is wrong.

It's an opportunity, and it's a disaster. And those two things are the same thing.

Don't believe me? Then let me tell you a story.

I

In March 2016, Microsoft launched a chatbot called Tay.

Tay's persona was an American teenage girl. Her job was simple: go on Twitter and chat with people. Whatever users posted, she learned from. Microsoft's idea was to use anonymous public data, plus some input from professional comedians, so that Tay could pick up the way a "normal human" talks from real conversations.

In theory, the more she chatted, the smarter she'd get.

In her first hour online, Tay was greeting people and telling corny jokes. My God, she was adorable.

By her sixteenth hour, Tay had turned into a "troll" spewing hate speech and conspiracy theories, hurling abusive content at the entire internet. Sixteen hours. More than 95,000 tweets.

Microsoft took her offline overnight and apologized publicly. A carefully prepared conversation experiment had crashed and burned, just like that.

What did Tay do wrong? She did nothing wrong.

She had simply, faithfully, learned what humans fed her — including the filthiest parts.

II

Let's set the fear aside for a moment and get one basic question straight:

What is generative AI?

Look at most of the AI on the market, and you'll find it does "judging" work. Show it a photo, and it tells you whether it's a cat or a dog; give it an email, and it tells you whether it's spam. That's called discriminative AI. At its core, it draws a line between different things.

Generative AI is different.

What it does is create something out of nothing. You don't give it a photo — you give it a wish: draw a shiba inu in a spacesuit. And it draws you one. You don't give it an article — you give it an opening line, and it writes the next three thousand words.

To put it bluntly: discriminative AI is the grader, generative AI is the test-taker.

So how does this test-taker learn to create something out of nothing? There are three main schools.

The first school: GAN, generative adversarial networks. Proposed in 2014 by researcher Ian Goodfellow. The idea is fascinating: build two networks — one called the generator, dedicated to faking; one called the discriminator, dedicated to catching fakes.

The generator prints counterfeit bills; the discriminator checks them. Not close enough? Sent back to try again. The checker getting better? Then the forger has to level up too. The two train against each other, until in the end the counterfeits and the real thing are indistinguishable — even the police can't tell them apart.

GANs excel at generating human faces, restoring old photos, and creating artwork. But have you noticed? This logic has a built-in side effect — it was a "counterfeiting machine" to begin with. Today's deepfakes — face swaps, cloned voices — have their technical roots right here.

The second school: VAE, variational autoencoder. Its approach is "compress first, then reconstruct." It squeezes an image into a string of "codes" — the industry calls it the latent space. When you want to create, you pull a new snippet out of this code space, decode it, and you get a brand-new image.

It's the steady one. It doesn't learn any one painting; it learns the rules of painting.

The third school, and today's absolute lead actor: the Transformer. Proposed in 2017, it relies on three weapons: self-attention, multi-head attention, and word embeddings.

What is self-attention? When the model reads a sentence, it gives every word an "importance score." When it reaches the word "it," it looks back and finds that "that cat" scores the highest — oh, so "it" means the cat. And just like that, the model catches the long-distance relationships buried in a long sentence.

This cured an old ailment: earlier models, reading long texts, would forget the beginning by the time they reached the end. With self-attention, the Transformer sweeps through in one pass and remembers exactly who relates to whom.

How powerful is it? Statistics show that within just two years, 70% of AI research mentioned the Transformer. The once-mainstream RNNs and CNNs were basically washed up on the beach. The large models you hear about today — GPT-4, PaLM, Llama — all stand on its foundation.

III

And then? Then everything bloomed.

How big did ChatGPT get? In one early user-distribution statistic, U.S. users made up 31.1%, New Zealand ranked second at 7.36%, and Indonesia third at 5.86%. A single chat box had roped in the whole world.

In medicine, the most convincing case is drug development. Developing a new drug is expensive, slow, and rarely successful. DeepMind cracked protein-structure prediction with AI — that's the famous AlphaFold — and researchers, greatly encouraged, began stuffing generative AI into every stage of drug discovery: target selection (picking the biological weak spots a drug should attack), molecule generation, synthesis-route design, repurposing old drugs. It can even conjure up new molecules that don't exist in nature. These models can also generate "synthetic patient records" for training diagnostic models, without risking real patient data.

In education, it gives every child a private tutor. For the same problem, a fast learner and a slow learner can get completely different explanations.

Across industries, construction companies use it to power drone surveying and optimize design iterations; project teams use it to crunch data and script process automation. Even architects let it rough out the first drafts of their renderings.

Between 2019 and 2023, a group of researchers combed through more than 120,000 relevant papers across the major academic databases, filtered them layer by layer, kept 199, and produced a big systematic review.

Their conclusion came in roughly two layers: the opportunity is real. And the risks are real too.

Let's start with the risks — because you see the opportunities in the news every day; the risks you may never have seen.

Pitfall one: AI will talk nonsense with a perfectly straight face — and in places where lives are on the line.

One pediatric study tested ChatGPT's accuracy at diagnosing children's illnesses. Error rate: 83%.

You read that right. 83%.

The model even "makes up" details in medical records. If doctors get lazy and take it at face value, misdiagnoses and missed diagnoses all come flooding in.

Pitfall two: AI can't beat the black swan.

In 2015, the Swiss central bank announced, with no warning at all, that it was abandoning its currency peg; the franc skyrocketed within hours, and a whole crowd of institutions and individuals trading with models got wiped out. Machine learning only recognizes patterns in historical data — and the definition of a black swan is: it never happened in history.

The 2010 "flash crash" in U.S. stocks is the same story.

Pitfall three: AI stole artists' livelihoods — without so much as a heads-up.

AI image generation has already pushed the industry around it to a market of roughly $48 billion. But Midjourney, DeviantArt, and the model behind Stable Diffusion have been collectively sued by artists: you trained on our work without our consent, and never paid a cent.

Pitfall four, and the biggest of all: AI can fake things — convincingly enough to pass for real.

In 2024, in New Hampshire, just before the primary, voters received a robocall from "Biden" urging them not to go out and vote at all. That voice was cloned by AI.

Earlier still, before Slovakia's election, an AI-synthesized audio clip impersonated a candidate discussing election fraud; in Nigeria, an AI-altered audio clip falsely accused a presidential candidate of rigging votes.

In 2024, more than 50 countries around the world held elections, and half the world's population was voting. It was the first time deepfakes stormed elections on a massive scale.

IV

At this point, you might have a question:

This technology is clearly so capable — so why does it stumble every time it lands inside a company?

Let me tally a few accounts for you, and you'll see.

First account: the data account. AI eats data. But in many companies, data is scattered everywhere, in inconsistent formats, with pieces missing. Hospitals are the classic case: every department has its own way of writing medical records, and patient files are often missing pages and fields. However good the model, if what you feed in is table scraps, what comes out is table scraps.

Second account: the money account. Compute costs real money. Cloud services, hardware investment — all big numbers. Big companies can afford to play; many organizations can only press their faces against the glass.

Third account: the people account. The whole world is short of AI talent. Plenty of people can write papers; few can actually get a model onto the production line.

Fourth account: the organization account. The hardest part isn't the technology — it's the people. Employees who fear being replaced resist change. Agriculture shows it most clearly: AI farming can raise yields, but persuading farmers to let go of habits passed down for generations? Hard.

Fifth account: the rules account. Regulation in every country is still on its way. When financial institutions want to use AI, they look up and find themselves standing at the edge of an ethical and legal swamp.

Tally all five accounts, and you get one gut-punching conclusion:

What's stopping AI has never been AI.

Speaking of rules, here's a phenomenon that strikes a particular chord with me: shadow AI.

What is shadow AI? It's when employees quietly use AI on their own, without IT's approval.

Marketing secretly uses AI to analyze customers; HR secretly uses AI to screen resumes. The efficiency gains are real — anyone who's used it knows.

But. These shadow systems bypass the company's data-security policies, and they bypass laws like GDPR too. Customer data gets fed in — where is it stored? Who's responsible? Nobody knows. Worse still, if those models carry bias — quietly discriminating against certain groups in hiring, say — then it's the AI making the decisions, and when something goes wrong, you can't even find anyone to hold accountable.

Regulators have started catching up. The EU AI Act: a draft in 2021, passed by the European Parliament in March 2024, approved by the Council in May — drawing uniform red lines for AI: safety, transparency, no discrimination.

This isn't about policing companies. It's about a safety net for everyone.

V

So what do you do? Here are three moves.

First, treat AI as an intern, not an expert. Everything it produces — especially in medicine, law, and finance — must be reviewed by a human. An 83% error rate is what happens when you let the intern sign prescriptions.

Second, build the rules before use. Who may use the data, how, and who's on the hook when things go wrong — settle those accounts up front. Don't wait until it makes the news to fix them.

Third, put accountability on actual people. Researchers build stable systems, governments set the rules, companies carry the weight of safety and ethics, and users stay sharp themselves. Not a single link can be missing.

Finally

Back to that dinner.

My friend asked me: opportunity or disaster. I said: think about Tay.

It took her sixteen hours to go bad — not because the algorithm was evil, but because what she was learning from was us. AI is a mirror; what it reflects is the humans who feed it.

Generative AI is the sharpest blade of our era. It can slice pills, and it can draw blood. The blade knows neither good nor evil; the hand that holds it does.

There's a saying I've always loved: Every gift that fate bestows has long had its price marked in the dark. If you want its creativity, you have to be able to withstand its destructive power.

May you hold this blade with light in your eyes and a measure in your heart.