New post on Eth Research!
Snappy with a memory: ~40% less gossip traffic
By:
- Nashatyrev
🔗
ethresear.ch/t/26078
Highlights:
- Compressing the entire gossip stream with shared context (instead of per-message Snappy) significantly reduces inbound bytes: ~73% of today’s bytes for a default node and ~63% for an all-subnets node using LZ77-in-Snappy-format with a 64 KiB history.
- Most of the gain comes from cross-message redundancy: the repeated topic string and repeated AttestationData across a slot effectively become short back-references, leaving an attestation largely as signature + indices (~117 bytes).
- A 64 KiB-history, Snappy-format LZ77 stream compressor gets big wins on consensus gossip: attestations compress to ~42% and aggregates to ~63% of current wire size in both default and all-subnets scenarios.
- The approach is practical to integrate: it can be added as an optional negotiated protocol (e.g., /snappy-stream/meshsub/1.2.0) that decompresses back into the existing gossipsub RPC stream with today’s ssz_snappy payloads—no changes to message IDs, validation, or the rest of the gossip stack required.
- Performance and safety look acceptable: decoding is essentially as fast as current Snappy plus a history buffer (same decompression-bomb bound); live tests showed large reductions on attestation-heavy streams (~43–44% wire bytes in a hub-and-spoke setup), while CPU and memory costs are predictable (64 KiB history per peer direction plus tables; compression is costlier than decompression but still a few percent of a core across several peers in rough estimates).
ELI5:
Normally, each gossip message gets squeezed (compressed) by itself, like packing each toy into its own small box. This research asks: what if we pack a whole stream of messages together, so we can reuse patterns we’ve already seen? Because many gossip messages repeat the same pieces (like the same topic name and the same attestation details), keeping a short “memory” of the recent past lets the compressor say “same as before” instead of sending the full data again. That can cut a lot of network traffic, especially for nodes that listen to many attestation subnets.