TL;DR. Two forward passes can share every MoE weight yet route the same token through different experts. We separate the resulting change into a pure routing term and a dense-like content term, then follow the routing term from block output to residual stream to behavior. Across seven checkpoints and two domains, raw routing magnitude varies much less than residual exposure; exposure is ordered by the routed block's share of the residual. Causal patches show that the routing term is real but bounded and largely reproduced by matched-norm noise. Router movement locates change. It does not establish behavioral influence.
Same weights can still mean different computation
Same-weight self-distillation creates an unusual comparison. A demonstration-conditioned teacher and a query-only student use the same parameters, but the added context changes hidden states and gates. The same token can therefore activate a different sparse computation in the two passes. A gate-overlap score can tell us that the route moved; it cannot tell us how much of the induced perturbation survives the rest of the network.
Teacher pass
Demonstration + query
Same model weights, but a worked demonstration changes both hidden state and expert gates.
Shared MoE block
Router + experts
The comparison holds parameters fixed and replays matched token prefixes through both contexts.
Student pass
Query only
The observed output difference combines changed gates and changed expert content.
The setting is deliberately single-step and same-weight. Accumulated optimization and separately parameterized teachers are outside this study.
Separate routing from content before measuring either
The routed-output change admits an exact add-and-subtract decomposition. The first term changes gates while evaluating expert outputs at the student's hidden state. The second holds teacher gates fixed while changing expert content. This prevents a common attribution mistake: calling every teacher-student block difference a routing effect.
routed difference = A (gate change at fixed content) + B (content change at fixed gates)
Switch between A and B to see what the decomposition holds fixed.
Gate change, content fixed
What did the changed routing assignment contribute?
Evaluate the teacher-student gate difference on the same captured student expert outputs. This is the term traced into the residual stream and patched behaviorally.
The identity is verified with a reconstruction gate for every included checkpoint. It is an accounting instrument; causal claims require intervention.
Routing magnitude is not residual exposure
We measured seven open-weight MoE checkpoints on PubMedQA and GSM8K, with 200 matched prompts per domain. The architectures span top-2-of-8 to top-8-of-128 routing, shared and shared-free blocks, and multiple gate conventions. The normalized routing term occupies a relatively narrow observed range, but its exposure to the residual stream varies twice as widely in multiplicative terms.
0.085-0.139
routing term as a share of routed block output across checkpoints and domains
1.6×
span of the block-normalized routing factor
3.2×
span of residual exposure, from 0.019 to 0.061
Backbone share orders exposure across the seven primary checkpoints: Spearman 0.96 on PubMedQA and 1.00 on GSM8K. The relationship is descriptive here; the next experiment manipulates the backbone directly. Select the figure to open it at full resolution.
The inversion is easiest to see at the extremes. Qwen3-30B uses the most fine-grained router and has one of the largest routing factors, yet the smallest residual exposure. OLMoE has a similar routing factor but the highest exposure. What differs is how much routed-block output contributes relative to the always-on residual backbone.
The backbone contains mismatch through routing coherence
Correlation is not mechanism, so we scaled the always-on shared output in DeepSeek-V2-Lite and Gravity while leaving the router untouched. Reducing this backbone increased post-block exposure monotonically: from 0.0245 to 0.0395 in DeepSeek and from 0.0403 to 0.0769 in Gravity. A frozen-numerator dilution control stayed nearly flat.
Measured exposure rises against a nearly flat denominator-only null. At matched input drift, preserving shared mass changes routing less than matched-norm noise, and both change it less than deleting the mass. Select the figure to open it at full resolution.
Mechanistic result
The always-on path does more than enlarge the denominator after a perturbation forms. Preserved common-mode mass helps keep the teacher and student routes coherent, containing the mismatch before it propagates.
Behavioral influence requires a behavioral test
The decomposition and exposure screen still do not prove that the recorded routing term changes model behavior. In a preregistered three-model test, we patched the captured routing term into every MoE block and compared it with the natural context effect and matched-norm random directions.
3.1-4.0%
greedy-token flips caused by the full routing term
40-45%
of the natural context effect recovered by the routing patch
58-74%
of the routing flip rate reproduced by matched-norm noise
The routing term has a real effect, but much of it is generic to perturbation magnitude. The content term tells a different story: its premium over a random direction is 5.6-80×, compared with 1.54-2.05× for the routing term on the same checkpoints and domain. Changed conditioning influences output primarily through direction-specific content, not through router movement alone.
A practical audit protocol
Step 1
Reconstruct
Verify that the A/B decomposition exactly reconstructs the routed-output difference for the checkpoint and execution path.
Step 2
Measure exposure
Quantify the routing term relative to the residual stream instead of inferring influence from overlap or total variation.
Step 3
Intervene
When a downstream decision matters, patch the recorded term or run another behavioral intervention. Exposure is a screen, not a threshold.
Durable lesson
Use router divergence to locate change, residual exposure to quantify propagation, and behavioral intervention to establish consequence.
Scope and limitations
The seven primary checkpoints are an opportunistic, family-correlated sample rather than a factorial architecture sweep. Backbone rescaling is confirmatory for two models, and behavioral patches cover three models and next-token effects rather than task-level generation quality. Scale probes and a merged-expert checkpoint show that the narrow block-level range is not universal. Most importantly, the claims concern one same-weight conditioning mismatch; they do not cover accumulated optimization, separately parameterized teachers, reinforcement-learning trajectories, or forgetting.
Interpretation boundary. Containment is not harmlessness. Exposure within the observed range is evidence about propagation in this regime, not a safety guarantee or a universal behavioral threshold.
Citation
BibTeX
@article{caruzzo2026routing,
title={Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation},
author={Caruzzo, Cedric and Yoo, Donggeun and Kim, Tae Soo},
journal={arXiv preprint arXiv:2608.15787},
year={2026}
}
