A mixture-of-experts (MoE) model splits its knowledge across many smaller "expert" networks and uses only a few of them for each word it produces. That makes it much faster than a single giant network of the same size, but it still needs memory for every expert. Meta's Muse Spark Open, released on October 15, 2026, is a clear example: 816 billion parameters in total, but only about 42 billion working on any one token.
This guide explains how mixture of experts works, why so many large open-weight models use it, and what it means if you want to run one yourself.
How mixture of experts works
A standard, or dense, language model sends every token through every parameter. A 70-billion-parameter dense model does 70 billion parameters' worth of work for each token it reads or writes.
An MoE model replaces some of those layers with a set of parallel expert networks and a small router (also called a gating network):
- A token arrives at an MoE layer.
- The router scores every expert for that token and picks the top few, called top-k routing.
- Only the chosen experts process the token, and their outputs are combined using the router's weights.
- The next token is routed again from scratch and may use entirely different experts.
The idea goes back to Shazeer and colleagues' 2017 paper, "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer". It showed that a router could keep most of a very large network idle for each example, so capacity could grow far faster than compute.
Total vs active parameters
MoE models are described with two numbers, and you need both:
| Model | Total parameters | Active per token | Experts |
|---|---|---|---|
| Muse Spark Open (Meta, 2026) | 816B | 42B | 8 of 256 per token |
| Mixtral 8x7B (Mistral AI, 2023) | 46.7B | 12.9B | 2 of 8 per token |
The Muse Spark Open figures come from its model card and Hugging Face's launch post. The Mixtral figures come from Mistral AI's announcement. Model names often encode both numbers: "816B-A42B" means 816 billion total and 42 billion active.
- Total parameters decide how much memory the model needs, because every expert must be loaded.
- Active parameters decide how much work each token takes, and so roughly how fast it runs.
Why MoE models are fast but memory-hungry
Generating text on a GPU or Mac is usually limited by memory bandwidth: for each token, the hardware has to read the weights it uses. A dense 816B model would read 816 billion parameters per token. Muse Spark Open reads about 42 billion, which is why a Mac with an M3 Ultra and 512 GB could run a 2-bit build at about 18 tokens per second in Hugging Face's test.
Memory is the other side of the trade. The router can send the next token to any of the 256 experts, so none can be left on disk without slowing generation sharply. The launch post says all experts must stay in memory: the model needs the memory of an 816B model and runs at roughly the speed of a 42B one.
You can see both effects in the LLM VRAM Calculator. Enter the total and active parameters separately: the memory figures follow the total, and the speed ceiling (with your memory bandwidth) follows the active count.
Dense vs mixture of experts
| Dense model | Mixture of experts | |
|---|---|---|
| Parameters used per token | All | Only the chosen experts |
| Memory needed | Total parameters | Total parameters |
| Speed per token | Follows total size | Follows active size |
| Compute per token for a given total size | Higher | Lower |
| Best fit | Small and mid-size models, limited memory | Very large models on machines with plenty of memory |
At the same memory budget, a dense model and an MoE model need similar hardware. The MoE model runs faster on it and can hold more knowledge for the same compute. When memory is tight, a smaller dense model is often the more practical choice.
What MoE means for running models at home
- Check total parameters against your memory. A model "with 42B active" does not fit a machine sized for a 42B dense model. Use the total.
- Expect good speed once it fits. If the whole model fits in fast GPU or unified memory, an MoE model generates text much faster than a dense model with the same total size.
- Quantize to fit. Lower-bit builds store every parameter in fewer bits. A 2-bit build of an 816B model is still about 265 GB of weights, so quantization helps but cannot make a huge model small.
- Avoid partial offloading when you can. Some apps can keep part of a model in system RAM or on disk. With MoE, any expert can be needed at any moment, so offloaded experts slow every token that uses them.
Model pages sometimes give sizes in GiB, the binary unit that is about 7% larger than a GB. The Data Storage Converter converts between them before you compare with your RAM.
Common questions
What does "A42B" mean in a model name? It is the number of active parameters per token. "816B-A42B" means 816 billion parameters in total, of which about 42 billion are used for each token.
Is a mixture-of-experts model smarter than a dense model? Not automatically. MoE lets a model hold more parameters for the same compute per token, which often helps quality, but results depend on training and data. Compare benchmarks for the specific models.
Does a mixture-of-experts model use less memory? No. It needs memory for all of its experts, the same as a dense model with the same total parameter count. It saves compute and time per token, not memory.
How many experts does a token use? It depends on the model. Muse Spark Open picks 8 of 256 experts per token, while Mixtral 8x7B picks 2 of 8.
How do I know if an MoE model will fit my computer? Multiply the total parameters in billions by the bits per weight and divide by 8 to get gigabytes, then add room for context. The LLM VRAM Calculator does the math for every quantization.