The Inference Bottleneck: Who Captures the Value?

IOSG Ventures

@IOSGVC · Jun 9

1,550

Author: @fraaaankfu | IOSG Ventures

Follow the GPUs: The $200B Question Becomes $600B

In 2023, Sequoia's David Cahn named the question hanging over the entire AI build-out — the $200B question. For every $1 spent on a GPU, roughly another $1 goes to powering it in a data center, so each year of GPU CapEx implied the chips would eventually have to generate on the order of $200B in revenue just to pay back the capital. Even on generous assumptions about AI revenue, he found a $125B+ hole between what was being spent and what end customers were actually paying. The worry was blunt: GPUs were being overbuilt ahead of real demand.

A year later the gap had widened, not closed. Cahn's 2024 follow-up reframed it as the $600B question as hyperscaler CapEx ballooned. The bear case hardened into a familiar shape — overbuild leads to a supply glut, and a glut incinerates capital.

Both pieces were really asking one thing: what fills the hole? The answer never arrived on the training side of the ledger. It arrived on the inference side — and the market only started to price it in over the past few weeks.

Cerebras IPO and the Inference Squeeze

Cerebras went public on Thursday, May 14. The IPO was 20x oversubscribed and priced nearly double Wednesday's final markup. Demand wasn't driven by a bet on the next Nvidia killer. It was driven by something simpler: the market waking up to the fact that inference, not training, is the real bottleneck in AI.

Cerebras' claim to fame is a chip architecture that makes inference incredibly fast. Not training — inference. That's what got Wall Street excited. The inference market is recurring. It scales with usage. Every time Claude answers a question or an agent executes a task, it costs compute. Training happens once. Inference never stops.

J.P. Morgan sizes the inference market at 10 to 50 times the size of training. When machines start running tasks dictated by other machines — agentic expansion — inference demand stops scaling with user count and starts scaling with compute itself.

Nvidia Redraws the Map: Inference Is the Headline

If Cerebras was the market waking up, Nvidia's latest earnings was confirmation straight from the top of the stack. On its latest earnings call, Jensen Huang said the quiet part out loud — AI demand is now growing parabolically. The reason is simple: agentic AI has arrived. Mainstream AI has moved from one-shot inference, to reasoning, to agents that call tools and orchestrate tasks on their own. "Tokens are now profitable," Huang said. In the AI era, compute is revenue and profit.

That reframes the whole industry. Training is the one-time cost of building a model. Inference is the recurring cost of running it — and inference, not training, is now the bottleneck.

Nvidia wrote that thesis into its own accounting. It now reports two platforms instead of one: Data Center and Edge Computing. Data Center (~$75B in the quarter, +92% YoY) splits into Hyperscale (~$38B, +12% QoQ) and ACIE — AI cloud, industrial, and enterprise (~$37B, +31% QoQ). The brand-new line is Edge Computing: $6.4B, +29% YoY, covering the endpoints where agentic and physical AI actually run — PCs, workstations, AI-RAN base stations, robots, and cars.

Edge is still under 8% of revenue, yet Nvidia promoted it to a co-equal "second platform." The signal is that inference is splitting into two fronts: cloud inference in the data center, and endpoint inference at the edge, where AI has to see, move, and act in the physical world. The roadmap follows the same logic — Vera Rubin, shipping from Q3, delivers up to 35x the inference throughput of Blackwell, and Huang put a fresh $200B TAM on the Vera CPU built for agentic workloads. Every frontier lab is expected to move to it on day one.

When the most valuable company on earth restructures its financial reporting around serving tokens, the bottleneck question is settled. The rest of this memo is about who captures the value once inference — not training — becomes the scarce resource.

Cerebras IPO and the Inference Squeeze

Cerebras went public Thursday. The IPO was 20x oversubscribed and priced nearly double Wednesday's final markup. Demand wasn't driven by a bet on the next Nvidia killer. It was driven by something simpler: the market waking up to the fact that inference, not training, is the real bottleneck in AI.

The Squeeze Is Already Here

Anthropic is the canary in the coal mine. Usage grew 80x beyond what they provisioned. Reports of Claude being "lobotomized" flooded the internet — throttled responses, slower inference, reduced context windows. Anthropic had to partner with SpaceXAI to use the entire compute capacity at Colossus 1: 220,000+ Nvidia GPUs and over 300 megawatts of power.

That capacity unlocked a rapid sequence of limit changes, each one a tell. On May 6, Anthropic doubled Claude Code's five-hour limits, removed peak-hour throttling, and raised Opus API rate limits sharply. On May 13, it lifted Claude Code weekly limits another 50% (through July 13). Then, from June 15, it did the opposite of generosity: it carved agentic and programmatic usage — Agent SDK, headless claude -p, CI pipelines — out of flat-rate subscriptions into a separate metered credit pool ($20–$200/month, billed at API rates). That last move is the whole thesis in miniature: agents consume inference at a rate flat subscriptions were never designed to sustain, so it has to be priced as the recurring cost it is.

Training is a one-time capital expense. Inference is a recurring operating cost that compounds with every new user and every new agent.

The Stack: Six Layers, One Bottleneck

Every AI application sits on a supply chain that starts in a TSMC fab and ends at an API endpoint:

Layer 0: Silicon → Server Assembly → Data Centers (Nvidia → Dell → Equinix)

↓

Layer 1: GPU Rental — CoreWeave, Lambda Labs, AWS

↓

Layer 2: Multi-Cloud GPU Aggregation — Modal, Together AI, Hyperbolic

↓

Layer 3: Model Deployment (Inference) — Serverless & Managed Endpoints

↓

Layer 4: Model APIs — Claude, GPT, Llama, OpenRouter

Most companies own one layer. Nvidia owns silicon. CoreWeave owns bare metal. Together AI owns inference optimization. OpenRouter owns model API routing.

Except one

Hyperbolic: The Only Company Spanning Three Layers

Hyperbolic launched its on-demand GPU marketplace in June 2025. By December, it hit $15M ARR with 200,000+ developers. Customers include Anthropic, Perplexity, and DoorDash.

The interesting part is the architecture.

Hyperbolic doesn't own a single GPU. Every card comes from neoclouds and data centers — CoreWeave, Lambda Labs, Nebius, smaller operators with surplus capacity. This sounds like a weakness. It's actually the moat.

By sitting between GPU suppliers and GPU consumers, Hyperbolic sees real-time data that nobody else has. It knows who's buying what GPU, at what price, at what time. It sees supply gluts before they become public. It sees demand spikes before they hit the market.

The company's stated plan is to use this data to train proprietary AI models that predict GPU pricing curves — then deploy its own capital as a GPU market maker. Same playbook Citadel used to dominate equity markets, applied to physical compute.

This is a flywheel:

Exchange generates data
AI models predict pricing
Better data → smarter routing today, pricing models over time
Better liquidity and prices → more developers → more clouds want in

No other company is attempting this. Hyperbolic is the only company that spans the GPU rental layer, the deployment layer, AND the model API layer simultaneously.

The Venice Parallel

Venice runs on two tokens: VVV (staking and platform access) and DIEM (inference credit). Projects stockpile DIEM to service inference. It's a clean model — but Venice is an application. It buys compute from the same constrained supply everyone else depends on.

What Venice actually sells is privacy. It isn't privatizing public compute into private property; it's wrapping commoditized inference in guarantees — no data retention, nothing used for training, requests anonymized, and some workloads run inside TEEs so even the operator can't see the plaintext. The underlying compute is a commodity; the privacy layer is the markup. And that guarantee is tiered, not uniform: for the open models Venice runs on its own or TEE-backed GPUs it can get close to end-to-end confidential compute, but for anonymized passthrough to closed frontier models (Claude, GPT) privacy means stripping identity only — the frontier lab still processes the raw prompt. The strongest privacy covers the open-model share; the frontier share is "anonymous," not "confidential." Venice's gross margin is the spread between its subscription price and the inference it pays for downstream, and the part it can charge above raw API cost rests almost entirely on that privacy premium — which is also why the margin is thin and exposed to frontier passthrough pricing.

Hyperbolic sits one layer below. If Venice is the gas station, Hyperbolic is the refinery. It aggregates fragmented GPU supply, standardizes it, and sells it to anyone. As inference demand grows, value accrues not just to applications that consume compute, but to platforms that aggregate and route it.

The fundamentals are healthier than the headline. The "$70M ARR" figure circulating publicly is almost certainly an artifact of counting subscription renewals as net-new acquisition; the defensible observable range is closer to ~$6M–$15M ARR. The error is mechanical: the $70M math takes the daily count of on-chain subscription transactions, assumes every one is a brand-new customer, and annualizes it. But subscriptions renew monthly, so most of that daily flow is the existing base re-upping, not net-new sign-ups. Read as renewals, the same daily count implies an active paid base in the low tens of thousands and revenue in the single-digit millions — an order of magnitude below the headline. Underneath, the traction is real: ~136,000 holders, ~9.9M monthly site visits (~330K/day), and new Pro subscriptions hovering around the ~1,400/day line. It's a genuine business — but a thin-margin one whose economics are hostage to the compute it buys.

That's exactly why Hyperbolic sits one layer below. If Venice is the gas station, Hyperbolic is the refinery. Venice buys compute from the same constrained supply everyone else depends on; Hyperbolic aggregates that fragmented supply, standardizes it, and sells it to Venice and everyone like it. As inference demand grows, value accrues not just to the applications that consume compute, but to the layer that aggregates and routes it — and captures the cost of revenue those applications pay out.

Why This Matters Now

Cerebras' IPO proved the market understands inference is a bottleneck. Anthropic scrambling for capacity proved it's a real problem. Agentic AI will multiply demand by orders of magnitude.

And it closes the loop on the $600B question from the other side. Cahn's bear case — overbuild, then glut — may well prove right eventually. But a glut is precisely the regime an asset-light aggregator wins in: when GPU prices fall and supply fragments across dozens of clouds, the player that owns no hardware and routes every workload to the cheapest available card captures the spread, while the operators holding depreciating GPUs eat the loss. Hyperbolic is long the glut, not short it.

The companies that win won't be the ones with the most GPUs. They'll be the ones that can tell you which GPUs are available, where, and at what price — and then put their own capital to work bridging the gap.

Hyperbolic is building that company. No GPUs. Pure software. Three layers deep. A Citadel for compute.