This is the last piece of the set.
Three quants (FP8, NVFP4 LoopShield, NVFP4A16) plus the EAGLE-3 head.
What are you running Nanbeige for? and what tok/s are you getting with the quant + draft head?
genuinely curious. Get em here:
ALT Four quants to run Nanbeige-4.2-3B
1
9
It also scales. single-stream it's fast, but aggregate throughput climbs to ~3,100 tok/s before the GPU goes compute-bound around 64 streams.
ALT EAGLE-3 stays net-positive with ~0.39 acceptance. Fast at every load.
1
1
9
Looped transformers (this model) reuse their weights each token.
Two things go sideways:
- quant error doubles
- the draft-layer naming clashes with the loop.
had to handle the virtual depth explicitly
huggingface.co/NullSense/Nan… Curious about people's speed numbers!
1
1
9
