Google Swapped AI Search's Dispatcher for a Tiny Model: R4T-Diffusion

A couple of days ago I came across a post on Google's official blog introducing a new framework called R4T-Diffusion. My first reaction after reading it: this is seriously good.

You may never have heard the name. But if you've ever used AI search, you've already been on the receiving end of everything it does.

Let me first walk you through the moves.

What is query fan-out?

You ask AI search a question — say, "help me figure out where to take my family this weekend."

The question is broad. AI doesn't just take that one sentence and search with it. It automatically splits your big question into a dozen, even dozens of smaller ones: kid-friendly attractions nearby, the weather forecast, the budget per person, family-friendly restaurants...

Once it's split, it searches them in parallel, then merges everything into a single answer.

This move — turning one question into a swarm of questions — is called query fan-out. Picture an extremely capable assistant: the boss gives one sentence of instruction, the assistant breaks it into a pile of tickets, hands them out to different people, and finally stitches the results into one report.

The problem is, dispatching is expensive.

The wider you split, the more ground the search covers and the better the answer. But more splitting also means burning more compute and waiting longer. It's a stubborn dilemma: either split broadly and well but crawl along like a snail, or be fast and split carelessly.

What Google did this time was replace the dispatcher.

How did they replace it?

The paper's approach, broken open, is three steps.

Step one: use reinforcement learning to train a large model so it knows what a good split looks like.

Step two: collect the high-quality splits that large model produces into a batch of samples.

Step three: train a tiny model with only 53.9 million parameters to imitate the large model's behavior.

53.9 million parameters — what does that even mean? In an era of models with hundreds of billions of parameters, this is a featherweight.

This playbook of "the large model demonstrates, the small model imitates" has a name: knowledge distillation. In 2015, Jeff Dean — then at Google — was among those who helped pioneer the path: transfer a large model's abilities to a much smaller one, so the small model can do eighty percent of the large model's work at a fraction of the cost.

Put plainly: the master demonstrates once, the apprentice quietly picks it up, and from then on the work gets done without ever troubling the master.

What counts as a good split?

You might ask: on what grounds is the small model's splitting better than before?

The paper sets three criteria for a "good split," which I'll translate into plain language.

First, no blind questions. Every sub-query it produces has to correspond to something findable in real databases. It can't spit out questions nothing in existence could answer.

Second, no duplicates. The model uses a metric called the Vendi Score to measure diversity, forcing it to spread its semantics wide enough. You can't split out "nearby attractions," "attractions close by," and "fun places around here" — those are three synonyms, not three questions.

Third, no drifting. Every sub-query has to be tethered back to your original big question; you can't split and split until you've wandered off into the clouds.

No blind questions, no duplicates, no drifting. Those three rules are the heart of this framework.

How much faster is it, really?

Let's do the math.

According to the paper, the old generation method worked like a queue, with one question popping out after another. When batches got big, latency climbed to nearly 50 seconds.

50 seconds — think about that. You type your query into the search box, brew yourself a cup of tea, come back, and it still hasn't finished splitting.

R4T-Diffusion, by contrast, generates every target sub-query in one shot, holding latency between sub-second and a few seconds. The official claim: 12 to 20 times faster than autoregressive approaches — the latency bottleneck smashed straight through.

12 to 20 times faster, while saving a hefty chunk of compute. That's the confidence behind the words "production-ready."

Is it only good for search?

No.

The paper explicitly lists where this framework can be put to work: recommendation systems, like Google Discover and YouTube's recommendations; creative search; exploratory information seeking.

Push the thinking one level deeper. The paper's authors believe this approach — using reinforcement learning to synthesize alignment training data — can extend to structured generation tasks where answers are inherently vague or subjective: planning, design, creative generation.

And that's where it gets interesting. Wherever there is no standard answer, there may be a use for it.

Is it live yet?

This is the part I care about most — and the part I'm least sure about.

The timeline is interesting: the paper came out in March this year; the blog post didn't appear until September 15 — a full six months in between.

Google's longstanding style is to only write it up with fanfare once the thing is already live. A six-month gap like this very likely means quietly testing, slowly rolling out.

There's a bit of circumstantial evidence too. Recently on social media, some people said they noticed more links being shown in AI Mode, some said their traffic went up, and some guessed it was an unannounced update.

But these are all guesses, with no hard evidence.

The one sentence the paper says that the blog doesn't

There's one detail I want to pull out on its own.

The blog post sings "production-ready" and "expert-level search" from start to finish, all applause. But at the end of the paper, the researchers added something the blog left out: R4T performs well in domains like fashion and music, yet in sensitive contexts it can amplify bias. To deploy it, you must first run domain-specific bias audits, paired with oversight mechanisms.

The paper's point: R4T is a tool, and beyond the tool there still has to be human judgment and human guardrails.

I love that warning. The more capable the technology, the more it needs someone holding the reins.

Finally

Back to the scene at the start: you ask AI a big question, and behind the curtain it quietly splits it into dozens of small ones, looks them up in parallel, and hands you a beautiful answer.

Before, the cost of "quietly" was steep. Now, one small model can make it fast and good at the same time.

Turn expensive intelligence into cheap instinct. This may be the best bargain in the AI business.

As for whether Google has already put it to work? And why it stayed silent six months ago?

I don't know the answer either. But next time AI search hands you a faster answer with more links, you can wonder whether some 53.9-million-parameter little model just split thirty questions for you.

That's my hunch. It may well be wrong.