A pile of legacy metal with 3x the memory bandwidth that costs 2/3 of the DGX Spark:
- v100 32GB 900 GB/s: $650 (x8)
- poweredge r720: $300 (x4)
~> $6500
bonus: it doubles as a space heater.
So you want local AI and want to know what hardware you need?
To get close to real-world, "cut the cord" local AI you need 256GB of memory in a decent package - can be DGX Sparks (TP-2), a Mac, or dedicated GPUs. It will cost you ~$10K USD or more depending on where you are and you can run GLM 5.3 Flash, Deepseek 4.1 Flash, or Qwen 3.8 Flash Next and many folks will be pretty happy. It's not going to be crazy fast but it will be usable - point OMP (or your favorite harness) at it and party on.
Caveats:
- even the best available model recipes at this level are not frontier equivalent - they're frontier-1 (or -2) at best, but most practical tasks and coding no longer strictly requires frontier
-the models often barely fit, need to be carefully optimized, and unless on dedicated GPUs ($$$) will just barely hit acceptable real world speed and concurrency for one person doing serious stuff or a small handful of people doing light stuff
- those boxes are gonna be busy and dedicated to model serving - little or no room left for sidecars < this is something that bit me and I ended up needing a 3rd spark 😂
That's not to say the next step down (in the 96GB-128GB - single spark - range) isn't useful. There are lots of very smart models you can run in that tier (and frankly on even smaller machines) that will do a lot of useful things. It's an an awesome place to start!
I've been there personally - first with a Macbook 96GB that I've been through model hell and back with, then with my first Spark that performed admirably in a solo configuration on several workloads for months. Both of them did real work 24/7 in complete privacy and were awesome learning platforms, but neither ever came close to being able to host a model to replace my daily driver.