🚀 Launch today: Kog generates 3,000+ output tokens/s per single request, on standard datacenter GPUs. We are bringing real-time LLM inference to hardware that companies already run in production. The speed previously associated with purpose-built silicon is now delivered on NVIDIA H200 and AMD MI300X. Today, we are opening our Tech Preview with a 2B coding model, with large frontier MoE support coming next. Try our Playground → playground.kog.ai 💥 Why that matters, and how we did it → blog.kog.ai/real-time-llm-in… 📖 Monokernel deep dive → blog.kog.ai/building-a-singl… 📖 Delayed Tensor Parallelism research → blog.kog.ai/delayed-tensor-p… read the thread 👇

LLM inference spends most of its time computing tokens on the GPU, but in-between tokens, the GPU has to wait for the CPU to schedule the next step. vLLM put a number on this last week as they removed two host syncs during one decode step, cutting the decode step by 17 percent! Their fix targets only a single step. The Kog Inference Engine (KIE) takes the host off the whole decode. The full generation runs as one persistent GPU program (or kernel), with no more scheduling from the host: the host starts it once, then goes quiet until the last token. If your batch-1 decode runs slower than it is supposed to, it is because time is lost between kernel launches!

LLM inference is slow enough that teams design whole products around the wait. Coding agents show it most plainly, though the same shape turns up in app generation and research loops. Background execution, review queues, and notify-me-when-done all answer the same question: where to put the user while the model finishes. Standard stacks generate 100 to 300 output tokens/s per request for a 2B model on high-end datacenter GPUs. Our 2B preview on the Kog Inference Engine (KIE) generates 3,500 output tokens/s per request on an 8× AMD MI300X GPU node, and 2,100 on an 8× NVIDIA H200 GPU node. That is 7× to 35× the baseline. Which of those three would you keep if inference ran 10× faster?

We are running the Kog Inference Engine (KIE) live on 8x AMD MI300X GPUs, 3,000 output tokens/s per request, and we are happy to talk through it in person. If your agentic loop spends its wall-clock time waiting on decode, that per-request speed is the reason to stop by. It keeps the loop moving at the pace of the engineer driving it. If you are working on agentic inference here, we would like to meet you.

During LLM inference, grid synchronizations across the GPU account for 6 occurrences per layer. 35% of every token's generation time was lost to that overhead. We reduced it by 9x. The standard approach uses a global counter. Each compute unit arrives, increments, and waits until the counter reaches 256. Every sync triggers full cache write-backs and invalidations across HBM, even when only a few values are needed. Measured cost on AMD MI300X between 7.6 and 7.9 µs per sync. Kog encodes readiness directly into the data. Buffers are initialized to NaN. Each CU polls only the values it actually needs, using scope-controlled loads that bypass unnecessary cache movement across chiplets. When the NaN disappears, the data is ready. Zero global counter contention. Zero broad cache invalidation on the critical path. 0.80 to 0.93 µs instead of 7.6 and 7.9 µs. Same hardware. That headroom goes directly into token generation speed. One of the reasons the Kog Inference Engine (KIE) generates 3,000 output tokens/s per request on MI300X. Full implementation with code, chiplet topology details, and the complete monokernel breakdown at blog.kog.ai/building-a-singl…

👀 Watch out @cerebras and @GroqInc - mystery model outputs 3000+ tps on standard GPUs. 🔥 Here is a comparison between Kog Laneformer-2B & Google Gemma3n-4B, both non-reasoning with same prompt. Laneformer @ 3000+ tps finished in 3s and Gemma 3n 4B took 43s.

3,000 tokens/s inference speed pulls developers in. Our launch last week proved it. Our post hit the Hacker News front page and stayed for 12 hours. 13,800 engineers read the Kog Labs technical breakdown. 2,240 developers tested our live playground, with a whooping 75% activation rate. More than 4 million tokens generated across thousands of conversations at an average generation speed of ~3,200 tokens/s. When inference is fast enough to feel different, developers come and build. Read our technical blog posts and test it by yourself. Try the playground → playground.kog.ai 💥Why 3,000 tokens per second matters and how we got there → blog.kog.ai/real-time-llm-in… 📖 Deep dive into the monokernel architecture on AMD MI300X → blog.kog.ai/building-a-singl… 📖 Delayed Tensor Parallelism, our approach to removing inter-GPU communication overhead → blog.kog.ai/delayed-tensor-p…

Incredible!

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. @Kog__AI just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

The monokernel idea was one of their powerful trick. Instead of launching many small GPU programs for normalization, attention, feed-forward layers, sampling, and communication, Kog keeps the whole decode loop inside 1 long-running GPU program. With a monokernel, weights for the next stage can start loading while the current stage is still finishing, so the GPU behaves more like a pipeline and less like a machine constantly being paused and restarted. If a Transformer layer is broken into many small GPU programs, the system can burn a scary amount of its budget just stopping, starting, syncing, writing, reloading, and waiting, before doing useful token generation. The monokernel tries to remove that stop-start behavior. Once it begins, it stays resident on the GPU and handles the full sequence, including prefill, decode, sampling, tensor-parallel communication, reductions, and internal state, without going back to the CPU for every little step. The big gain is that weight streaming stays continuous. For batch-size-1 inference, the GPU mostly needs to stream active model weights from high-bandwidth memory into compute units as smoothly as possible. Read more about their “monokernel” implementation here. blog.kog.ai/building-a-singl…

Today, we're releasing LFM2.5-8B-A1B, a device-optimized model designed to power real-life applications on phones, laptops, PCs, robots, and fast & lightweight server-side use-cases. > 8B MoE, 1.5B active > Expanded 128K context > LFM2.5 flagship hybrid MoE architecture > Trained on 38T tokens + large-scale RL > fast, reliable tool calling, punching above its weight, comparable to models with up to 4x its size > customizable on a single GPU for any specialized task > LFM2 open-weight license 🧵

Kog officially launched today! Super-fast AI inference speed on standard GPUs, 30x faster than ChatGPT. And it's European deep tech. Check it out 👇