What is MoE?
Mixture of Experts is a model architecture where the neural network consists of many specialized sub-networks (“experts”). A router network decides for each individual token which experts are activated. The remaining ones stay inactive.
Core idea: Large knowledge (many parameters), fast inference (few active parameters).
How it works
Input token
↓
Router network (gating function)
↓
Selects k out of N experts
↓
Only the k experts compute the token
↓
Results are combined via weighted sum
↓
Output token
Example gemma4:26b: 128 experts + 1 shared expert. Per token, only 8 out of 128 are activated.
MoE vs. Dense: Comparison
| Property | MoE (gemma4:26b) | Dense (gemma4:31b) |
|---|---|---|
| Total parameters | 25.2B | 30.7B |
| Active parameters per token | 3.8B | 30.7B (all) |
| Experts | 128 + 1 shared | none (one large network) |
| Inference speed | Faster | Slower |
| VRAM requirement | ~16 GB | ~20 GB |
| Quality (benchmarks) | Slightly below Dense | Somewhat better |
Important: Although only 3.8B parameters are active per token, all 25.2B parameters must reside in VRAM. The router needs to be able to access any expert at any time.
When to use which architecture?
MoE recommended
- Multimodal tasks (images + text)
- Deep analysis with thinking mode
- Very long context (256K+)
Dense better
- Pure text workflows (claims, descriptions)
- Parallel multi-agent pipelines
- Deterministic outputs
gemma4 variants at a glance
| Variant | Type | Active params | Context | Target device |
|---|---|---|---|---|
| E2B | Small | 2.3B | 128K | Mobile devices |
| E4B | Small | 4.5B | 128K | Laptops |
| 26B | MoE | 3.8B active | 256K | Workstations |
| 31B | Dense | 30.7B (all) | 256K | Workstations |
Thinking mode in gemma4
gemma4 is a thinking model: it can internally “think” before answering. In practice:
- Normal requests: Thinking disabled (
think: false), prevents empty outputs with structured prompts - Deep analysis: Thinking enabled, allows more thorough analysis, but takes 2-5 minutes
Note: gemma4 uses a different disable mechanism than qwen3.5. The
think: false API parameter is required. The /no_think prefix from qwen3 does not work with gemma4.