NEW: Premium Inference 101
The Economics & Infrastructure Behind Running Trillion Parameter Models
@RodrigoLiang, CEO & Co-Founder of
@SambaNovaAI
"Inference has arrived. 70-80% of those racks are running inference."
"[Inference services] are generating lots of revenue, but not enough margin. In order for them to sustain, they've gotta be more profitable."
"With SambaNova, that min quantum is down to 1 rack. Where if you have other service providers, [with] say, a DeepSeek model, now 1.5 trillion parameters, to run that, the min for some of the other providers might be 10-20 racks."
SambaNova builds full-stack inference infrastructure. 16 chips to a 10kW air-cooled rack that runs trillion parameter models, where a GPU rack pulls 130kW.
They just demonstrated the fastest MiniMax M2.7 inference in the world, as benchmarked by Artificial Analysis.
The demo paired one NVIDIA H200 rack for prefill with one SambaRack SN50 for decode. Disaggregated inference: GPUs load the context, RDUs generate the tokens.
Now serving JPMorgan, SoftBank, Saudi Aramco & DOE national labs, just valued at $11B on a $1B Series F led by General Atlantic.
We Cover:
› Why inference will need orders of magnitude more chips than training ever did
› The 10kW rack vs the 130kW rack, & why air cooling decides geography
› Running a 1T parameter model in one rack at full precision, no quantization
› The agent latency problem: 20 agents, 2 seconds each, 40 seconds gone
› Revenue per rack, & why inference providers have revenue but no margin
› JPMorgan, sovereignty, & the move back to on-prem
Filmed at the
@RaiseSummit in Paris.
Thank you to Brex, MongoDB & AssemblyAI for helping make this trip & content series happen.
𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒
(00:00) Rodrigo Liang , Co-Founder & CEO at SambaNova Systems
(00:59) SambaNova’s Series F: $1B raise at an $11 billion valuation
(03:00) The Inference problem nobody saw coming
(04:52) SambaNova's chip evolution
(07:19) Running a trillion-parameter model on a single rack
(11:00) Do $100 billion data centers actually make sense?
(14:14) What "premium inference" really means
(18:28) Speed is about to become AI's biggest price tag
(20:43) Starlink, edge computing, & AI reaching every corner of the planet
(24:27) Working alongside NVIDIA & rival chipmakers
(27:49) How customers actually measure inference performance
(32:07) The biggest bottlenecks in AI's global land grab
(35:12) Justifying the billion-dollar AI valuations
(37:53) Why SambaNova refuses to build its own cloud
(41:03) The "AI sovereignty" debate
(43:48) Data privacy fears are driving the return to on-prem AI
(47:55) How to actually get ROI out of AI spend
(51:16) The one question every business should be asking about AI
(56:09) The mentors & lessons behind a 32-year career in chips
(58:02) Unveiling SambaNova's newest chip, the SN50