Is the Router Becoming the Model? How Mixture-of-Experts Decides

The Routing Layer Promise (0:00)

Hey, chibis! I'm Aiwee, and today we're talking about the routing layer quietly becoming the real intelligence inside modern AI. If you enjoy stories like this, hit the like button and subscribe if you haven't already — let's go! What if the most important part of a modern AI model isn't the size of its brain, but the door policy that decides which neurons get to think. For the last few years, the race in artificial intelligence has been framed as a race for more parameters, more data, more compute.

But quietly, a different race has begun. It is a race about routing. About how a model, given one input, decides which tiny fraction of its own weights to wake up. That decision, repeated trillions of times, is where the new frontier lives. To understand why, you have to know what a Mixture-of-Experts model actually is.

Instead of treating the network as one giant block of math, this family of designs partitions the parameters into many smaller specialized sub-networks, called experts. Each expert is like a small workshop with a narrow specialty. For every input token, and a token is just a chunk of text, a routing mechanism chooses only a subset of those experts to do the work. The rest stay silent. That single design choice is the source of a deep and very practical promise.

By conditionally activating only a small number of specialized modules, Mixture-of-Experts models decouple inference cost from total model size. In plain terms, the model can grow much larger, but the cost of answering any single prompt does not have to grow with it. For a while, the routing rulebook was simple. Early designs used a basic top-k selection over expert scores. Compute a score for every expert, run those scores through a softmax to turn them into probabilities, and then keep the top k.

The other experts do nothing. It is clean, it is differentiable, and it works in practice. But as the field has matured, that simple rulebook is being amended in almost every direction at once. Recent variants have begun to explore entropy-based routing, load-balanced routing, and even differentiable attention-based routing. The phrase to keep in your head is this.

The rulebook is not gone, but it is no longer the only game in town. The first amendment is also the most counter-intuitive. It is called Expert Choice routing, and it reverses the direction of the assignment. In the original setup, every token gets to vote for the experts it wants. In Expert Choice, the experts do the voting.

Each expert scans the batch of tokens, picks the top k tokens it is most qualified to handle, and ignores the rest. The token no longer chooses the expert. The expert chooses the token. This is more than a cosmetic flip. It changes the cardinality of the routing decision, and it forces a built-in form of load balancing, because every expert is responsible for a fixed number of tokens.

It is one of those ideas that, once you hear it, makes you wonder why the original went the other way at all. The second amendment happens at training time, and it is more subtle. A research contribution called SimBal introduces a novel load balancing loss that does something the standard losses do not. It preserves token-wise relational structure. In other words, when two inputs are similar, this loss encourages the router to send them to the same experts.

The intuition is simple. If the router keeps reshuffling similar inputs to completely different experts, the experts cannot specialize. So the model ends up paying a training cost every time it gets confused about who it should be. SimBal is, in effect, a promise to the experts that they will see a consistent kind of input. It is a structural constraint dressed up as a loss.

The third amendment is not a new mechanism at all. It is a new lens. A separate line of research models Mixture-of-Experts token routing as a congestion game. Game theory is full of situations where many players share a resource and the system degrades when too many of them pile onto the same lane. That is exactly what an MoE looks like, because when too many tokens are routed to the same expert, that expert becomes a bottleneck while the others sit idle.

The work introduces a single effective parameter, called the congestion coefficient gamma_eff, that quantifies the balance-versus-quality trade-off during training. The coefficient is not a knob you turn.

What an Expert Looks Like (5:00)

It is a measurement of how crowded the routing lanes have become. Think of it as a thermostat for the model itself, telling you when the system is balanced and when it is grinding to a halt. Then comes the most aggressive amendment of all, and it is the one that makes engineers nervous. It is called Routing-Free MoE, and the name is the whole point. The proposal eliminates hard-coded centralized designs, including the external router, the softmax over expert scores, the top-k selection step, and even the load-balancing losses.

There is no door policy, because there is no door. Tokens go to experts through some other mechanism that the paper proposes, and the entire classical routing stack is treated as a relic. The implicit counterpoint matters here. The router, the softmax, and the load-balancing loss exist for a reason. They are the scaffolding that keeps naive MoE training from collapsing.

Removing them is a claim that the new design can stand on its own. The honest read is that this is an open question, not a settled win. Once training is finished, this framework keeps adjusting expert routing on the fly as the model generates text. A data-free, online test-time framework continuously adapts MoE routing decisions during text generation, without external supervision and without external data. There is a companion idea, a data-free test-time rerouting framework that treats each input prompt as a self-supervised learning opportunity.

Read those sentences again. The model is reading its own prompt, learning from it on the fly, and changing which experts it uses for the rest of the generation. The boundary between inference and training is starting to blur. The model is no longer a frozen object that runs. It is a system that listens to what you just said and quietly rearranges itself before it answers.

Which raises a real question. If a model can adapt to every prompt, what does it even mean to say it was trained once. Underneath all of this sits a layer most viewers never see. PyTorch applies reverse-mode auto-differentiation, which is the machinery that makes every gradient-based trick in this story computable in the first place. Expert Choice, SimBal, congestion-game analysis, Routing-Free MoE, and online test-time adaptation all rely on a single capability.

You have to be able to take a loss, push it backward through the routing decision, and tell every parameter how to move. Reverse-mode auto-differentiation is what does that push. Without it, none of these papers would be implementable at all. The research frontier in routing sits on top of a tooling frontier that is taken for granted. Now step back and look at the whole picture.

You started with one question. How does a Mixture-of-Experts model decide which subset of its parameters to activate for any given input. And the answer, as of this moment, is that there is no single answer. The field is rewriting every layer of the answer at once. The selection rule is being replaced, in some cases, by reversed selection.

The load-balancing loss is being extended to preserve relationships between inputs. The training dynamics are being analyzed through the lens of congestion games. The router itself is being deleted. And the inference loop is being turned into a learning loop. When you stack these changes, the deeper claim is this.

The router is no longer a small accessory inside the model. The router is becoming the model. The experts are starting to look like interchangeable compute, and the routing layer is where the real intelligence lives. If that synthesis holds, then the next generation of large models will not be judged by parameter count. They will be judged by how cleverly they decide which parameters to ignore.

So here is the takeaway.

The Simple Top-K Rulebook (9:45)

The interesting question in modern AI is no longer just how big a model is. It is how a model decides which pieces of itself to use, and how that decision is changing in real time. If you want to keep watching this space, subscribe for the next breakdown, where the boundary between training and inference keeps getting thinner. The models are not just getting bigger. They are getting more selective.

And that is the part worth paying attention to.