I learned something new. Based on this, the memory bandwidth of DGX Spark is even lower. Looks like RTX 3090/4090/5090 or other graphics card with higher memory bandwidth is the way to go for higher tok/s.
My Mac is still useable for running overnight task for my local jobs. I’m not in a hurry to invest in a new machine.
11 tok/s on 8-bit is already near the M2 Max ceiling. 400GB/s over ~27GB per token is ~14 theoretical, so you are at about 75% efficiency, which is normal for decode.
4-bit halves the bytes per pass: ceiling ~28, low 20s measured is the realistic jump.
A 27B fits that machine fine, no Spark needed.