Hey, everyone.
Benchmarked my semantic caching gateway ( on 5 models: DeepSeek Flash V4, GPT-OSS-120B, Gemini 2.5 Flash, Claude Haiku 4.5, GPT Luna Pro.
p50 latency: 2.9-7.7s uncached, ~82ms on a semantic hit, ~1.4ms on an exact match.
98.5%+ lower p50. Code + benchmarks:
Link in the thread: