KVPrior: On the Suprising Effectiveness of Marginalizing KVCaches over Prefixes
To avoid costly prefills, non-prefix KVCache methods try to reuse individually prefilled KV caches in non-prefix positions. To recover the accuracy degradation, existing methods either perform...
zenodo.org