Two Ways to Tame Discontinuity: K3 and DeepSeek V4

Su Jianlin’s brief on K3’s MoE and Attention is easy to summarize: SiTU, pre-up-projection Norm, QB, four MLA constraints, NoPE. That is already a solid engineering note.

Repeating it adds little. The motivations and ablations are already there.

The question I want to ask is different:

If DeepSeek V4’s three defenses are scaffolding for a representation manifold that was never smooth, can K3’s Stable LatentMoE and Attention trade-offs be read in the same coordinate system?

This post proposes a unified reading: one diagnosis, two prescriptions. That is not a claim that the K3 and V4 labs share a geometric consensus. It is a decision to place both stacks inside the three-axis frame from my tear series and ask what lines up and what is merely a difference in design.

Prior posts in that series: DeepSeek V4 and Manifold Tearing, How Torn Is a Trained MoE?.

Evidence is kept in three layers: official facts (K3 / V4 reports and architecture tables), Su’s reading, and my geometric reading. The last two should not be treated as conclusions endorsed by the labs.

1. One chart of symptoms (a reading, not a verdict)

Large-model training failures often look like three unrelated incidents:

1. Activation blow-ups. When both SwiGLU branches align with the input, multiplicative amplification between the gate path and the linear path produces outliers and loss spikes.
2. Routing glitches. Top-k expert choice is discrete; tokens on opposite sides of a boundary can induce an $O(1)$ jump in block output — a genuine $C^0$ tear, not a steep slope.
3. Small errors compound. A small upstream inconsistency, carried across dozens of residual layers, becomes a downstream loss spike.

In the V4 post I compressed these into three axes (my coordinate system):

Official narratives do not use this vocabulary. K3 talks about activation explosion, scale variation, and expert load imbalance. V4 is also candid that the principles behind Anticipatory Routing and clamping are not fully understood.
Folding both into three axes is a reading choice, not a paraphrase of lab self-description.

Architecture in one line (Su):

K3 = KDA + MLA + Stable LatentMoE + AttnRes

The optimizer remains Moonlight-style Muon; Muon updates for the attention projections are applied independently per head. Su stresses that heads should not be forcibly coupled in the update; the K3 report emphasizes balancing per-head update scale, improving large-scale training stability, and a modest cost win. I will say “decouple head update scales,” not “correctness theorem.”

2. Comparison table

Details below.

3. Activations: hard wall or soft rail

V4: hard clamp

V4 clamps activations and gates into a finite range.
My reading: a chart is only valid locally; outside it, first-order steps fail. Clamping says “do not leave the map.”

K3: SiTU softcap

Conventional MoE experts typically use SwiGLU:

Both linear branches scale with the input; after gating and elementwise multiplication you get multiplicative outlier activations (no need for a fourth-degree claim). K3 replaces SiLU with SiTU (Sigmoid Tanh Unit): it softcaps the gate to $(-\beta,\beta)$, then softcaps the linear branch as well — SiTU-GLU (Su: $\beta_1=4,\ \beta_2=25$).

Su reports that under the same bounds, softcap often beats the hard clips used in GPT-OSS and DSV4.

Incremental hypothesis

“Bound the activation” understates the shape of the boundary:

• Hard clip is continuous and 1-Lipschitz. It does not introduce a finite jump in function value. At the threshold it is typically not $C^1$; outside, gradients die — kinks and saturation, not a new $C^0$ tear.
• Softcap damps magnitude with a smooth cap; this creates fewer flat, zero-gradient regions and milder curvature.

Both say “stay on the map.” The difference is whether the edge is a hard kink with a saturation plateau or a graded guardrail.

So: compare saturation rates and gradient/curvature concentration near the bound; do not expect hard clip to show up as a function-value jump wall under a hardG zoom test. An earlier draft that called this a “secondary tear” picked the wrong topological object — that claim is withdrawn; the contrast stays.

4. Routing: discrete choice in a latent space

Standard MoE tears (measurement side)

Standard top-k: hidden state $h \in \mathbb{R}^d$, router scores; on the boundary where the k-th and (k+1)-th router scores meet, the block output can jump with a $C^0$ discontinuity. I measured that on released OLMoE and Qwen weights: the tear is real.

LatentMoE: change the decision space, do not remove discreteness

Structure (Su and the K3 report):

• Standard MoE: $d \to D \to d$, n choose k
• LatentMoE: project down, run a larger expert pool at higher top-k in latent space, project up

Official scale (do not confuse with intermediate experimental baselines):

Not “448→8, then double with sparsity held fixed.” If Su’s essay mentions figures around 448, treat them as design discussion or intermediate settings, not the K2→K3 architecture table. The sparsity factor moves from 48 to 56 — slightly sparser, not unchanged.

Cost: down-project, MoE, up-project in series makes scale and stability harder. K3 keeps an RMSNorm before the up-projection. The official report says this Norm consistently improves validation loss and downstream benchmarks — not “val loss about the same, only benchmarks move.” Su’s extra intuitions (routed/shared balance, weak nonlinearity as effective depth) sit in the interpretation layer.

Load: with a larger pool, the loss-free balancing scheme remains, but SignSGD-style updates get unstable at this scale. QB (Quantile Balancing) replaces them, approximating global quantiles with histogram bins (Su: ~1000 bins is enough).

What this means under the reading

1. The official K3 problem statement is explosion, scale, and imbalance — not “we cured top-k $C^0$ tears.” Reading Latent/QB as tear therapy is my extrapolation and must stay labeled as such.
2. Routing in a lower-dimensional latent space, with a larger pool and larger k → the M1/M2/M3 diagnostics from how-torn are worth re-running under Latent settings. That is a measurement sequel, not numbers this post already has.
3. Pre-up-projection Norm is a continuous patch beside a discrete routing block: it does not change top-k logic, but can change scale and division of labor.
4. QB is a large-scale traffic allocation problem — same family as fair allocation under capacity constraints. It need not be sold as geometric tear medicine.

V4’s Anticipatory Routing emphasizes temporal consistency of decisions; K3’s LatentMoE and QB emphasize spatial scaling and load balance. Different axes, both make discrete-expert systems trainable. Side-by-side is useful; merging them into one therapeutic target overclaims.

5. Attention: two routes, not just “sparse”

Four constraints that must hold together (Su)

1. Quality not worse than MLA
2. Train and prefill cost not above MLA
3. KV cache smaller than MLA (very long context)
4. Decode compute smaller than MLA (MTP / speculative decoding friendly)

Su’s judgment: no simple design currently hits all four. Under a KDA hybrid, some of MLA’s weaknesses ease, so K3 still keeps MLA.

Reading V4 attention: CSA + HCA, not vague “full-ish”

More precise labels (see the DeepSeek V4 architecture notes):

• CSA (Compressed Sparse Attention): sparse selection on compressed representations
• HCA (Heavily Compressed Attention): after heavy compression, no sparse selection — not “full-ish attention on the original sequence”

“Extreme sparse + compress” is a one-line summary; the two layer types have different jobs. Su’s high-level genealogy — DSV4 as “push MLA’s decode form to the extreme, then recover efficiency through compression and sparsity” — is useful for contrasting K3’s Linear+Full path; layer-type details follow the official docs.

NoPE: short official claim, geometric extension is mine

K3’s MLA drops RoPE. Hard facts:

• Dropping RoPE in a full-MLA setup usually hurts (K2 lineage)
• In a KDA+MLA hybrid, putting RoPE back adds almost nothing → drop it for simplicity

The K3 report supports a weaker claim: KDA supplies position-sensitive, recency-aware sequential mixing, so periodic MLA layers can use NoPE.

Su’s further discussion via PaTH / Householder / DeltaNet is suggestive.
My extra step (hypothesis, not official): read this as “positional duty can be partly outsourced to the linear branch’s dynamics,” and extend it into the language of geometric connections. Evidence stops at KDA’s sequential mixing; past that is coordinate-system extension, not a report sentence.

Two long-context routes (for contrast)

Which goes farther is open. The table is a coordinate system for the next tech report, not a bet.

6. What lines up under this reading, and what remains a difference in implementation

Aligned phenomena (reading layer — not a joint lab endorsement):

• Unbounded or overly aggressive activations destabilize training
• Discrete expert selection induces real discontinuity structure (whether labs list it as the primary target is a separate question)
• Cross-layer / inter-block transport needs stability patches
• Attention is a multi-objective constraint, not “newest module wins”
• Architecture patches are mostly scaffolding; whether data and representations are smooth to begin with remains another ceiling (V4 post, Max Ma’s cut — still open)

Recipe forks:

• Boundary: hard clamp vs softcap (kink/saturation vs smooth rail)
• Stability: mHC’s doubly stochastic residual mixing vs Latent pre-up-projection RMSNorm
• Routing: temporally consistent decisions vs latent scale-out + quantile balancing
• Long context: CSA/HCA vs KDA+MLA

Whether frontier labs “first treat the net as a geometric object, then constrain” is my interpretation, not an established collective consensus. What one can say: Muon, mHC, SiTU, QB, and Anticipatory Routing have different names and all add constraints to hard-to-train discrete/deep systems; placing them in one frame is useful.

Su’s method note still matters: minimal change, few new variables per step, ablations everywhere. That discipline matches “measurable diagnostics + minimal scaffolding” without requiring geometric vocabulary.

mHC, one tightened sentence

What holds rigorously: mHC constrains the residual-stream mixing matrix to the doubly stochastic set, so that mixing has spectral norm ≤ 1 (non-expansive). The full layer is still

The nonlinear branch $F$ does not make the full layer 1-Lipschitz by that constraint alone. In the table, write “constrain pure residual transport to be non-expansive,” not “inter-layer perturbations cannot grow.”

7. Not yet measured, but should be

1. Softcap vs hard clip: saturation and near-bound gradient/curvature — not hardG hunting for function jumps.
2. M1/M2/M3 on LatentMoE: how cliffs and boundary mass change with latent routing and a larger pool; effect of pre-up-projection Norm on/off on block-output scale and jumps.
3. Open-weight genealogy: whether available hybrid / compressed-attention weights support a coarse “two routes” picture.

Until there are numbers, the geometric reading is a coordinate system, not a verdict.

8. Conclusion

K3 and DeepSeek V4 need not be framed as mutually exclusive.

• V4 pushed training stability into a set of almost nameable engineering defenses.
• K3 presents MoE scalability and multi-objective Attention trade-offs as a set of understated, ablation-backed improvements.

Side by side: not two mutually exclusive “correct architectures,” but two scaffolds under similar engineering pressure — hard vs soft boundaries, temporal routing vs latent scale-out, interleaved compress/sparse vs Linear+Full hybrid.

Whether data and representations are smooth upstream is not settled on either side.
Which long-context route goes farther is not settled either.

What helps now is reading the next tech report with this author-proposed coordinate system in hand: ask first whether it constrains activations, routing, or cross-layer transport; then whether the patch is a hard constraint or a soft one; and only then look at the benchmark number.

Evidence layers: official — Kimi K3 tech report, Kimi-K3 architecture notes, DeepSeek V4 docs, mHC paper; interpretation — Su Jianlin, K3 MoE and Attention; coordinate system — DeepSeek V4 and Manifold Tearing and How Torn Is a Trained MoE?.