🔥 New @SarvamAI STT release — Saaras V4, with Pipecat support on launch day. 22 Indian languages + English. 10 of its Indian languages have no commercial alternative. Watch @VinayakGavariya @SarvamForDevs demo key features

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on @huggingface, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

Pipecat v1.8.0 today, with launch day support for Gemini 3.5 Transcribe, Google's new Gemini-based transcription model. This is a very big Pipecat release. There are 233 entries in the changelog. Some highlights: 🛡️Improvements to error handling and service failover: Processors can now report when they are no longer usable, errors are categorized, and ServiceSwitcher only fails over when a service really can't recover. STT/TTS services also stop endlessly retrying permanent failures like invalid API keys or models. ⚡ Faster pipeline startup: Pipeline setup has been reworked around the new setup() lifecycle, allowing processors and services to initialize and connect concurrently before the pipeline starts. Combined with import-time improvements, this significantly reduces startup time, especially for larger pipelines. 🗣️ More flexible turn management: Services with built-in turn detection now propose turn boundaries, while Pipecat's turn strategies make the final decision. This keeps turn management and interruption handling in one place and makes provider-native turn detection much easier to customize. 🧪 Pipecat Evals keeps getting more powerful: Run scenarios during development to iteratively measure pass rates, get machine-readable results.jsonl, evaluate turns individually, run entire directories of scenarios, and more. Great for measuring nondeterministic behaviors like interruptions and async function calls instead of testing them once and hoping for the best. 🤖 Better coding-agent experience: Pipecat Context Hub is now included with the CLI and integrated into pipecat init, including automatic setup for supported coding agents and freshness checks to help prevent agents from generating code against stale Pipecat APIs. 🔧 Function calls can now be cancelled by the LLM: Long-running async tools can opt into LLM-driven cancellation, and timed-out function calls are now automatically cancelled instead of continuing to run in the background. 🔌 New Speechify TTS, ElevenLabs Dialogue TTS, Bland TTS, Sarvam Realtime STT, Deepgram Flux on SageMaker, OpenClaw Gateway support, LiveKit runner support, MoQ client mode for connecting bots through a relay, including deployments behind NAT and more. And, of course, there are tons of fixes and smaller improvements throughout the framework, including pipeline startup/cleanup, metrics, TTS tracking, transports, realtime services, and turn handling. Huge thanks to everyone in the community for making everything we do possible! There are now more than 180 Pipecat integrations. ❤️ Happy hacking! 🚀

🙌

Congrats to @DeepgramAI on the GA launch of Flux TTS, now live in Pipecat 🎉 @JonPTaylor looks at how Flux TTS delivers a more consistent voice experience. Flux reads the whole conversation, not just the next line — adaptive tone, consistent pronunciation, clean interruption handling natively (no SSML markup, no style tags!) all at sub-200ms latency. ➡️ docs.pipecat.ai/api-referenc…

@aiDotEngineer Concierge is now live! Talk to the concierge and build your custom schedule for the next three days along with a list of people ( speakers ) who you should definitely meet! This was built on the go with @pipecat_ai and @GradiumAI Try it out: aie-concierge-web.onrender.c…

🔥 ⚽️ !!

My favorite annual AI event is next week in San Francisco. Come hang out with me at the @aiDotEngineer World's Fair. Go buy a ticket. Or, if you're a student, there's a great volunteer program. The organizing team does a great job making the World's Fair a conference with great talks, an opportunity for deep-dive learning across multiple AI engineering domains, a gathering of people building the infrastructure for AI adoption, and an opportunity to see early what everyone will be talking about next year. I'm doing a couple of talks. One on voice interfaces, with @neilzegh. And one about experimenting with new building blocks for "AI-native" software, software patterns for applications we couldn't have built at all before now and that we'll take for granted in a few years! I get more requests than I can keep up with, via LinkedIn and email, to meet and talk about AI engineering, voice AI, and @pipecat_ai. Most of the year, my default response is, "I would love to but I can't, if you make a PR I'll try hard to look at it." AI Engineer World's Fair is the one time of year when I can say: if you'll be at the World's Fair, come find me. We'll geek out about what you're building and all the crazy ideas we both have!

🔥 New STT model

🔥

👏🏽 Try them today with the Pipecat CLI

Microsoft announced a bunch of interesting new AI models and tools this week. Model launches alway get lots of attention. But don't sleep on the new ASSERT evals framework that launched today. I'm on record as arguing that 2026 is the year of evals. Evals are the glue for all the "jobs to be done" at every level of AI: model training; testing and deciding on what models to use and how to use them; and testing and improving AI agents in production. Evals unify our work on those different layers of the stack. These days, when we talk about evals, observability, and testing, we're talking about overlapping parts of a large set of tools we're still early on in figuring out. As the AI engineering ecosystem matures, diversifies, and increases massively in scale, we really, really need good evaluation (observability, monitoring, testing, data management) frameworks. I got a chance to test the new Microsoft ASSERT evals framework before it was released, and it has some very nice core ideas. 1) ASSERT is open in two important ways. First, the team is serious about broad support for models, frameworks, and use cases. Microsoft spent time understanding voice agent use cases and building Pipecat support, for example. Second, the code is completely open source, released under an open MIT license. 2) We're all working in and with agentic coding tools today. That means we are planning in natural language, and all of our software development and ops tools have to evolve for these new, natural language, workflows. ASSERT takes descriptions of desired agent behavior and generates specifications for the ASSERT suite of tools to run against. In a world where "English is the programming language," how we actually make natural language "code" precise enough and repeatable enough is perhaps the big unsolved tooling problem that all of us are working towards in different ways. This is true whether we work on coding agents, AI opps tooling, orchestration frameworks, or vertical applications. 3) Microsoft describes ASSERT as a policy-driven framework. Rather than eval against generic performance metrics, ASSERT aims to generate stable but adaptable evaluation criteria for specific agents. "Policy-driven" also implies a full loop design. Policy (generated from specific requirements) -> evaluation -> optimization -> monitoring in production -> improving the policy description -> evaluation -> ... 4) Enterprise agents need to be evaluated along many dimensions: task completion, individual conversation turn behavior, latency, mode-specific metrics like audio disfluencies, and safety/security. Microsoft designed ASSERT to be used together with a new safety governance toolkit called Agent Control Specification. 5) Finally, ASSERT is integrated into the Microsoft Foundry ecosystem. Today, AI engineering tools have to be open source and vendor neutral to get attention from developers and gain widespread adoption. *And* it's equally important to give enterprise customers tools that work as a coherent stack. This is hard to do well. There are real tensions between open source development versus engineering a great full stack developer experience. However, if you sweat the details on both ends, you benefit from a full spectrum of feedback about real-world development pain points. It's more work, but it's worth it! Kudos to Microsoft for embracing this and committing to an open, community oriented approach, plus doing the extra work to build the full stack for enterprise customers.

SF Voice AI Meetup livestream. Presentations and panels start at 7:15 Pacific. piped.video/live/EWys7ij9TTQ…

Local native-audio voice agent running on an RTX 5090. - @NVIDIAAI Nemotron 3 Nano - audio|text ➡️ text - patched vLLM to implement complete turn prefix caching - ~125ms TTFT - @kyutai_labs Pocket TTS - text ➡️ audio - Nemotron Speech ASR - streaming audio ➡️ text - @pipecat_ai Smart Turn end-of-utterance - ~500ms total voice-to-voice latency - runs bash via tool calls If you're interested in voice and realtime multi-modal AI, come join us at the SF Voice AI Meetup on Thursday May 7th. Talk to engineers from NVIDIA, Kyutai, and Pipecat about what you're building! Links to meetup registration, code, and models on @huggingface below ...

Voice AI Meetup, Thursday May 7th. This one's a special crossover event. T-Bot, who hosts the global Voice AI Spaces meetups, is visiting San Francisco and will MC! - NVIDIA researchers will present some of their really cool recent work on speech models. - We'll have demos and two fireside chats, featuring new developments in models and evals, with @GradiumAI, @ArtificialAnlys, @ServiceNow, and @pipecat_ai. - And, of course, 🍕 and great conversation. - Thanks to the @trychroma team for hosting in their wonderful office/event space. Registration link below. Come hang out with 150 old and new friends!

Excited to support the new @DeepgramAI Flux Multilingual model

Smallest AI is now natively supported in @pipecat_ai Lightning TTS + Pulse STT can now plug directly into your Pipecat voice agent pipeline. Docs below ⬇️

We just made Pipecat testing a lot easier. With @cekuraAi + @pipecat_ai , you can now get: • full traces • every tool call with inputs + outputs • complete transcripts with timestamps • mock tools so agents don’t hit live APIs • chat + WebRTC testing, all in one place Everything in one place for both test runs and production debugging. Docs below 👇