"Mixture-of-Depths Attention"
This paper teaches a Transformer to attend not just across tokens, but also to depth KV from its earlier layers.
That helps recover shallow-layer signals that standard residual stacking tends to dilute, improving performance with only a small extra compute cost.
Similar idea to Kimi’s Attention Residuals, but MoDA modifies the attention module itself, while AttnRes changes the residual/depth aggregation path.