← Back to Projects & Writing Hub

MoE Routing: From Movement to Behavioral Consequence

TL;DR. A router can change without telling us how much of that change reaches the model's residual stream, which expert-output directions receive the moved mass, or whether the final behavior changes at all. This program builds that missing measurement ladder. The aim is not to find one more routing score; it is to separate the questions that routing diagnostics often collapse.

Research-program diagram connecting MoE routing change, residual exposure, correspondence geometry, and behavioral verification
The program separates four questions that routing statistics often collapse: what moved, how much propagated, where the mass landed, and whether behavior changed.

Program thesis

Use router divergence to locate change, residual exposure to quantify propagation, correspondence geometry to diagnose where moved mass lands, and behavioral intervention to establish consequence.

Two complementary studies

Both studies compare two forward passes of the same MoE with the same frozen weights: a demonstration-conditioned teacher and a query-only student. They share the experimental backbone, but their primary estimands are deliberately distinct.

Two same-weight MoE passes decomposed into routing and content terms before residual exposure
Residual exposure study · arXiv:2608.15787

Routing Divergence Is Not Evidence of Behavioral Influence

An exact routing/content decomposition tracks a same-weight routing mismatch from the routed block into the residual stream, then tests its effect with causal patches.

Read the project page →
Moved MoE gate mass aligned with expert-output directions and compared with an exact relabeling reference
Correspondence study · Workshop paper under review

Which Expert Directions Receive Moved Gate Mass?

An exact conditional reference asks whether observed gate movement aligns unusually with captured expert-output directions, then tests whether that geometry allocates behavioral measurement.

Read the project page →

The measurement ladder

Question 1

What changed?

Separate gate-induced routing mismatch from the dense-like content shift created by changed conditioning.

Question 2

Where did it propagate?

Measure the routing term relative to the routed block and to the residual stream instead of treating gate movement as influence.

Question 3

Did behavior change?

Use geometry as a diagnostic and a behavioral intervention as the evidence when the downstream decision matters.

Program roadmap

Decompose and trace the mismatch

The residual-exposure study isolates the gate-change component and follows it through the block, backbone, and output.

Characterize expert correspondence

The correspondence study conditions on captured expert outputs and intact gate pairs to ask where moved mass lands.

Allocate expensive behavioral audits

An ongoing challenge formulation asks which layers and routing changes are worth measuring when interventions have unequal cost.

Integrate the ladder

Future main-conference work will connect these pieces into a broader account. Details will be added when that study is ready to be public.

What the program does not claim

The current results concern same-weight, conditioning-induced routing changes. They do not yet establish what happens across accumulated optimization, separately parameterized teachers, reinforcement-learning trajectories, or deployment-scale task outcomes. The studies also show why a single scalar proxy is unlikely to be enough: exposure does not order every behavioral effect, and correspondence geometry reverses as an allocator across checkpoints.

← Back to Academic Projects