How Torn Is a Trained Mixture-of-Experts?

A Geometric Diagnostic for Routing Discontinuity in Released Weights

A companion to DeepSeek V4 and Manifold Tearing. That post was the argument — loss spikes are geometric tears, not optimization blowups. This one is the measurement: I built a diagnostic, pointed it at released MoE weights, and asked how torn they actually are.

In the DeepSeek post I used a word that came up talking with Max Ma: a Mixture-of-Experts layer doesn't bend the representation, it can tear it. (The term itself is older than that conversation — it's Lee and Verleysen's, from their manifold-learning work on tearing topologically nontrivial data manifolds; Max Ma and Gen-Hua Shi's Deep Manifold papers motivate the neural-network manifold viewpoint but never use the word.) Bending is continuous and benign; tearing is discrete and the pathology. It was a good story. But a metaphor is a debt — at some point you have to show the tear exists, say how big it is, and admit what it does and doesn't cause.

So I paid the debt. What follows is a measurement paper compressed into a blog post: a geometric diagnostic, run on released OLMoE and Qwen weights, with negative controls and known-answer tests.

A metaphor I owed a measurement

A MoE layer routes each token to a small top-$k$ subset of experts. That discrete choice makes the layer-to-layer map $C^0$-discontinuous: two hidden states arbitrarily close, but on opposite sides of a routing boundary, go to different experts — so the block output can jump by an $O(1)$ amount. That jump is the tear.

Prior work approaches this from two sides and stops short of measuring it on real models:

• Continuous-routing fixes — ReMoE replaces top-$k$ with ReLU routing and explicitly frames top-$k$ as a jump discontinuity; Soft MoE and sparsemax/α-entmax are relatives. These remove the discontinuity; they don't quantify the one sitting in shipped weights.
• Routing flip-rate — the R3 router-replay study reports that ≈10% of routers and 94% of tokens flip at least one expert between the training and inference engines. That counts how often the expert set changes, not the output jump the change induces.

Nobody had measured the quantity that actually governs the transport map on a real model: the output-jump geometry on released LLM-MoE weights — how big the jump is, where it sits, and which direction triggers it. That's the gap.

What a tear actually is

A MoE block maps $h \in \mathbb{R}^d$ to

with router logits $g_e = W_e h$, top-$k$ selecting the largest, and $w$ the softmax over all router logits, gathered at the top-$k$ slots without renormalization (released OLMoE and Qwen1.5-MoE both ship norm_topk_prob=false). Order the logits $g_{(1)} \ge \cdots \ge g_{(k)} \ge g_{(k+1)}$. The $k/(k{+}1)$ active-set boundary is where

a hyperplane with normal $n = W_{(k)} - W_{(k+1)}$. Cross it and you swap the $k$-th expert, jumping the block output by $\alpha\lVert E_{(k)}(h) - E_{(k+1)}(h)\rVert$, where $\alpha$ is the boundary slot's gathered router weight — so the unweighted $\lVert E_{(k)} - E_{(k+1)}\rVert$ (the M2 cliff below) is an upper bound, while the actual $\alpha$-weighted size is what the per-block jump below measures. That is a genuine $C^0$ discontinuity — contrast a ReLU MLP, which is $C^0$-continuous with $C^1$ kinks.

So how do you tell a real tear from a steep-but-smooth slope? You zoom in. Walk a short path across the boundary, measure how far the output moves per step of input — the difference quotient $\lVert \Delta y\rVert / \lVert \Delta x\rVert$ — then look closer, and closer again (I sample at 500, then 2000, then 8000 points along the path). A smooth slope flattens as you zoom: finer steps, proportionally smaller jumps. A real tear does the opposite — it stays a sheer wall however close you look, so the measured steepness keeps climbing in lockstep with the zoom.

That's a clean yes/no test, and it hands you a number that is very easy to misread — so let me head that off now. I zoom by a factor of 16 (from 500 to 8000 samples), so a genuine tear's steepness grows by exactly 16×, while a smooth map barely moves. Call that growth hardG. The thing to notice: that 16 is just the zoom factor. Any real tear — at any layer, any $k$, any model — lands on 16×, because $16 = 8000/500$. So 16× does not tell you how bad the tear is. It tells you the tear is real — a wall, not a slope. How bad it is comes from a different number (the size of the jump, below). Two controls that are smooth by design keep the test honest — a tied-expert swap (a no-op) and a continuity-guaranteed soft gate — and both come back at $\approx 1\times$ (no growth), as they must.

Conceptually, the test is simple: smooth maps stay around $1\times$ as you refine the grid, while a true jump grows with the zoom factor. A sanity check on a toy block with an exact, known tear nails the law down: zoom in, and the steepness tracks the zoom factor perfectly — a clean linear law ($R^2 = 1.000$) across a 64× span of resolutions — while the smooth controls stay flat. That known-answer test is what tells us the 16× we see in real weights is the tear's fingerprint, not a quirk of the probe.

Three core measurements come out of this:

• M1 — boundary prevalence: the distribution of the margin $g_{(k)} - g_{(k+1)}$ and the fraction of tokens sitting near the boundary.
• M2 — expert cliff: the normalized $\lVert E_{(k)} - E_{(k+1)}\rVert$, plus the cosine of the swapped pair. cos ≈ 1 means redundant (no real tear); cos ≈ 0 means non-redundant at the $\sqrt{2}/2$ baseline; cos < 0 means genuinely specialized. This separates "outsized specialization" from "merely non-redundant."
• M3 — continuity signature: the zoom test above — does the steepness grow with zoom (a tear) or flatten out (smooth)? — summarized as hardG, with its two controls.

Severity, then, is not in the continuity signature. It is carried by the per-block jump and M2 — keep that split in mind for everything below.

The result: released routers are torn at every layer

The tear is not a toy-model artifact. On OLMoE-1B-7B (16 layers, 64 experts, $k=8$), the zoom test comes back positive at every layer — steepness growing 16× in lockstep with the zoom (hardG $\approx 16\times$, range 15.93–16.01) while both smooth controls stay flat ($\approx 1.0\times$). That's the measurement figure below. And remember the catch: that 16× just means "real tear, every layer" — it is not a measure of size. The size is two other numbers. First, the jump itself, $\approx 0.239$ — about 24% of the MoE-sublayer's output shifts when a token crosses the boundary (a per-block geometric quantity, not a change in the model's final answer, its logits, or task accuracy; as a perturbation added to the residual stream it is smaller still, consistent with the small downstream KL below). Second, the expert cliff M2 $\approx 0.70$ — cosine $\approx 0.025$, right at the baseline for two unrelated vectors, so the swapped experts really are different, but not wildly specialized. And almost every token sits within a hair of a boundary (median margin $\approx 1.5\times 10^{-3}$).

Table 1 — OLMoE-1B-7B per-layer diagnostic (24 texts). hardG is the zoom-test growth (steepness at 8000 ÷ steepness at 500); for a real tear it equals the zoom factor 16, so its flatness across layers is the test's doing, not the model's. The block-jump column is the per-block output jump (fraction of block-output norm). hardG, block jump, and the two controls are medians over 8 boundary-crossing paths; margin, M2, cos are per-token medians. Across layers: median hardG 15.99×, M2 0.704, cos 0.025; mean block jump 0.239; soft/tied controls ≈1.00.

Layer 15 (the last block) is the lone mild outlier — a lower cliff (M2 0.60) and higher cosine (0.30 → more expert redundancy) — yet its discontinuity (hardG 15.93×) is undiminished.

This is not an OLMoE quirk. Qwen1.5-MoE-A2.7B (24 layers, 60 routed experts, $k=4$) reproduces the same pattern:

Could the 16× just be an artifact of $k=8$? No — and now there's direct proof, three ways. The clean reason: 16 is the zoom factor (8000/500), so any real tear hits it regardless of $k$. The cross-check: Qwen routes $k=4$ and lands on the same 16× as OLMoE's $k=8$. And the direct test: I dialed $k$ through 1, 2, 4, 8 on the same model — the number never budged (hardG 16.00 every time, total range 15.92–16.03). It tracks the zoom factor, not $k$.

The one-line summary: training does not sew up the seam. Released routers carry the tear at every layer — the zoom test positive, the per-block jump real at ≈24%, the cliff M2 ≈ 0.70 sitting at the unrelated-vector baseline — on both families.

The tear is directional — and that's the surprising part

Here is where the naive intuition fails. The obvious robustness test is: perturb the hidden state randomly and see if removing the tear makes the block more stable. The answer is a null — and the null is informative.

Even at a perturbation large enough to flip 67.6% of tokens' top-$k$ sets, the hard-vs-soft block jump differs by only 2.6% (0.581 vs 0.566). Killing the tear barely changes the block's response to random noise. The null isn't that experts never flip — at this magnitude 67.6% of them do — it's that random flips add almost no discontinuous excess on top of the smooth response the soft gate already produces.

The reason is geometric: random perturbations are almost always tangent to the boundary. Perturb along the raw-logit boundary normal instead, at $2\times$ the per-token distance-to-tear, and the picture inverts:

The distance-to-tear is tiny — median 0.24–0.34% of $\lVert h\rVert$ — so the model lives on the boundary. A boundary-normal nudge of under 1% relative magnitude flips the expert ($\approx 0.9$ probability) and produces an $O(0.1)$ per-block output jump (a fraction of the block-output norm, not the model output) with measurable downstream KL, while an equal-magnitude random/tangent nudge does next to nothing. The tear is exploitable, but only along a specific low-dimensional direction.

And you can't cheaply re-gate it away. Continuous re-gating removes the static tear (softG $\approx 1\times$), but applied to all layers at inference it raises perplexity 10.04 → 159.94 (15.9×) at the gentlest threshold. Single-layer re-gating is mild ($\approx 1.2\times$). So a practical mitigation has to be layer-targeted — or, as DeepSeek-V4 does, folded into training rather than bolted on afterward.

Training: it decomposes, it doesn't detonate

This is the honest negative, and it's the part that keeps the metaphor from overreaching. To probe training I used a controlled from-scratch GPT-MoE probe at OLMoE-like geometry (E=64, k=8, ≈308M params) — not the 7B model's pretraining.

Mechanism (real, modest): a parameter-space difference-quotient probe finds hard routing rougher than a continuity-matched soft gate — 2.0× vs 1.4×. Directionally consistent with the synthetic 4.5×, but weaker as the geometry becomes realistic.

Outcome: natural training is spiky-but-convergent, and the tear does not self-heal. Over 8000 steps there are 196 spikes >0.3 (max 0.585) yet no divergence (final loss 3.27). M2 holds 0.711 → 0.690 and hardG stays $\approx 16\times$ throughout, while the operational whole-block jump collapses early (0.431 → ≈0.15) then sits in a noisy band. Training suppresses the tear's consequence, not its topology.

Push harder with a tear-magnitude dial across two seeds at the edge of stability:

No run diverges. Spike count isn't even monotone in tear level. But full tear reproducibly seeds a rare severe-but-recovered spike (max ≈4–5 vs ≈0.5). The honest statement: the tear contributes optimization roughness and rare spike severity, but is not by itself sufficient for collapse at this scale — momentum and Adam absorb it. Scale-dependence stays open, and that's exactly where DeepSeek-V4's engineering becomes relevant.

Reading DeepSeek-V4's three defenses geometrically

This is where it closes back on the DeepSeek post. The key caution: routing discontinuity is not loss spike. The discontinuity is the first of several separable factors:

1. Routing discontinuity — directional, the geometric entry point (§ above).
2. Expert-outlier magnitude — the jump scales with it; I measure the correlation between $\lVert h\rVert$ and the $k/(k{+}1)$-swap jump at +0.437 (high-norm tokens jump more).
3. Temporal backbone/router mismatch — untested here (future work).
4. Cross-layer propagation gain — downstream-KL / injected-jump $\approx 0.02\text{–}0.09$ (a non-expansive residual bounds it).
5. Optimizer absorption — the training result above.

DeepSeek-V4's three interventions map onto factors 2–4 — which I read as acting on the consequences, not the topology: SwiGLU clamping (factor 2), Anticipatory Routing (factor 3), and manifold-constrained hyper-connections (factor 4 — a residual map on the Birkhoff polytope, spectral norm ≤ 1, non-expansive). These are readings of DeepSeek's own described mechanisms — DeepSeek doesn't claim to be targeting the tear this post defines.

I can verify the first one directly. A SwiGLU-clamp sweep on released OLMoE leaves the topology flat — hardG 16.03 → 16.02, M2 0.705 → 0.701 — while reducing absolute amplitude (expert cliff 2.90 → 1.97, absolute hardJump 0.214 → 0.164); the relative jump stays scale-invariant (0.310 → 0.303). Clamp caps the jump's amplitude, not its existence — the DeepSeek decomposition, measured rather than asserted.

In the DeepSeek post I drew this table, arguing V4's four mechanisms all do the same kind of thing in different positions — admit that weights and activations are geometric objects:

The measurement here says the geometry those mechanisms respect is genuinely there, in the shipped weights — a real tear at every layer — whether or not the optimizer ever happened to trip over it during training.

What this is, and what it isn't

This is a measurement/analysis result, and I want to be precise about its boundaries:

• It does not propose a new gate, a new $C^0/C^1$ taxonomy, or claim first discovery of the discontinuity. ReMoE, Puigcerver et al. (2022), and the spline-theory line own those.
• What it adds: a diagnostic that runs on released weights with negative controls and known-answer tests; a cross-model characterization (OLMoE + Qwen); a directional inference result with a mitigation bound; and an honest training-time decomposition that refuses to overclaim.

The bottom line: the MoE routing tear is real, measurable, and cross-model — a true tear (the zoom test is positive at every layer; steepness grows with zoom, the smooth controls stay flat) that trained routers carry rather than sew up. It is severe in the jump itself (≈24% of the block's output) — not in the 16× (that's the zoom factor, not a size), and not in how specialized the experts are (M2 sits at the unrelated-vector baseline). Its inference consequence is directional: random inputs miss it, boundary-normal inputs of under 1% magnitude hit it, and you can't re-gate it away post-hoc for free. Its training role is a decomposition, not a single cause — which is precisely why the engineering remedies cap amplitude rather than remove the tear.

The DeepSeek post argued that treating the network as a geometric object is moving from philosophical stance to engineering default. This one supplies the number that the stance was missing.

Caveats, stated plainly. The zoom test only tells you whether there's a tear and where — not how big; its 16× is the zoom factor any real tear must produce, not a model property and not "bigger-is-worse." Every number here is a geometric quantity on hidden states: the ≈24% block jump and the boundary-normal fragility are not tied to end-to-end task accuracy (GSM8K, MMLU), and the jump is measured one layer at a time, so how tears compound across stacked blocks is untested. Beyond that: two model families and 24 short prompts; a small-scale 308M training probe (the spike-severity signal may be scale-dependent); a single-path hardJump trace (multi-path resampling would give error bars); clamp quality measured within a hook path; the temporal-mismatch factor named but not measured. Deep Manifold (Ma & Shi) is used as motivation only — external prior work, not my own framework. The numeric source of truth is a result-JSON set; every quantitative result figure regenerates from it with a stdlib-only SVG script.

Paper and code: github.com/waylandzhang

References

• Ziteng Wang, Jun Zhu, Jianfei Chen. ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing. ICLR 2025. arXiv:2412.14711
• Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Neil Houlsby. From Sparse to Soft Mixtures of Experts. ICLR 2024. arXiv:2308.00951
• André F. T. Martins, Ramón Fernandez Astudillo. From Softmax to Sparsemax. ICML 2016. arXiv:1602.02068
• Ben Peters, Vlad Niculae, André F. T. Martins. Sparse Sequence-to-Sequence Models. ACL 2019. arXiv:1905.05702
• Joan Puigcerver, Rodolphe Jenatton, Carlos Riquelme, Pranjal Awasthi, Srinadh Bhojanapalli. On the Adversarial Robustness of Mixture of Experts. NeurIPS 2022. arXiv:2210.10253
• Wenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, Fuli Luo. Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers (R3). 2025. arXiv:2510.11370
• Boris Hanin, David Rolnick. Complexity of Linear Regions in Deep Networks. ICML 2019. arXiv:1901.09021
• Randall Balestriero, Richard Baraniuk. A Spline Theory of Deep Networks. ICML 2018. arXiv:1805.06576
• Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, Yoshua Bengio. On the Number of Linear Regions of Deep Neural Networks. NeurIPS 2014. arXiv:1402.1869
• William Fedus, Barret Zoph, Noam Shazeer. Switch Transformers. JMLR 2022. arXiv:2101.03961
• Barret Zoph et al. ST-MoE: Designing Stable and Transferable Sparse Expert Models. 2022. arXiv:2202.08906
• Dmitry Lepikhin et al. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. ICLR 2021. arXiv:2006.16668
• DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026. arXiv:2606.19348
• Max Y. Ma, Gen-Hua Shi. Deep Manifold Part 1: Anatomy of Neural Network Manifold. 2024. arXiv:2409.17592
• Max Y. Ma, Gen-Hua Shi. Deep Manifold Part 2: Neural Network Mathematics. 2025. arXiv:2512.06563
• John A. Lee, Michel Verleysen. Nonlinear Dimensionality Reduction of Data Manifolds with Essential Loops. Neurocomputing 67, 2005. DOI:10.1016/j.neucom.2004.11.042
• Niklas Muennighoff et al. OLMoE: Open Mixture-of-Experts Language Models. 2024. arXiv:2409.02060
• Qwen Team. Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters. 2024. Model blog

Related: DeepSeek V4 and Manifold Tearing · The Four Realms of Neural Networks