What is a MoE (mixture of experts) model?
Updated
A mixture-of-experts (MoE) model splits parts of its network into many smaller "expert" blocks and uses a router to send each token through only a few of them. It can hold a large total number of parameters while doing roughly the work of a much smaller model per token: quick like a small model, but needing the memory of a big one. A dense model, by contrast, uses all of its parameters for every token.
Try it now, free, no account neededStart a story with VesperThe ship's AI is kind. And keeping a secret.Total and active parameters
Model sizes count parameters, the learned weights; what 12B means covers the basics. An MoE has two counts that matter. The total is every weight in the file. The active count is how many are actually used to produce each token. Names often show both, as in a label like "30B-A3B": thirty billion in total, about three billion active.
Inside, the model still has the usual stack of layers. In some of them, the single large feed-forward block of a dense model is replaced by a set of experts, and a small router decides which few experts each token visits. A layer might hold dozens of experts and use only a handful per token. Different tokens, even in the same sentence, can take different routes.
What "expert" does and does not mean
The word suggests one expert for history, one for romance, one for code. That is not how it works. The experts are not assigned topics; specialisation emerges during training and is often about patterns in text, such as punctuation, certain word types or a language, rather than subjects a person would name. There is no roleplay expert to switch on, and you can't choose which experts answer.
Dense vs MoE, side by side
| Dense model | MoE model | |
|---|---|---|
| Weights used per token | All of them | A small share |
| Memory needed | Matches its size | Matches its total size |
| Speed per token | Slower as size grows | Close to a model of its active size |
| Running at home | Limited by graphics memory | Often split between graphics memory and system RAM |
| Feel in long chats | Depends on training | Depends on training |
A rough rule many hobbyists use is that an MoE behaves somewhere between a dense model of its active size and one of its total size. It is only a rule of thumb: training data, tuning and the prompt matter more than architecture for how a character sounds.
Running one yourself
Every expert has to be loaded, because any token might need any of them, so an MoE needs memory for its full total. The upside is speed once loaded. Because only a few experts work per token, people running models at home often keep the shared layers on the graphics card and push experts into slower system RAM, and still get usable speed. Combine that with quantization and large MoE models fit on hardware that could never run a dense model of the same total.
Speed figures you read online assume particular hardware and settings, so treat them as examples, not promises.
Does it matter if you use a hosted model?
Mostly no. Through an API or a site you see replies, not architecture: what you notice is how fast the first words arrive, how well the character holds together and how much of the story the model reads. UnsavedAI serves Gemma 4 12B on every plan and Gemma 4 31B on Priority, listed on Models, and runs them on its own servers, so there is nothing to load or offload on your side. Judge any model by a scene you know well; what a roleplay model is suggests how. Vesper, a ship's AI with a secret, is a good test of whether a model keeps a hidden thread.
Questions people ask
Is a mixture-of-experts model better than a dense one?
Not automatically. It trades memory for speed. A well-trained dense model can write better than a larger MoE, and the reverse is also true.
Why is my MoE slow at home if only a few billion parameters are active?
Usually memory. If experts sit in system RAM or on disk, moving their weights becomes the bottleneck. More of the model on the graphics card helps.
What does A3B mean in a model name?
About three billion active parameters per token. The number before it is the total size of the model.
