AI systems are on pace to run more web searches than humans this year, and within a few years the ratio could hit 1,000 to one. @WilliamBryk, CEO of Exa, digs into what a search engine built for that shift actually looks like in "The Search Engine for the Agentic Web," posted by @aiDotEngineer on YouTube. The talk lays out why keyword search breaks down for agents, and what an API built specifically for LLMs needs to do differently. - Search built for the "guzzler." Human search is short, simple queries; he describes an AI agent's search behavior as an "information guzzler," and argues no single engine can be optimal for both humans and agents at once. - Google's blind spot. He points to typing "shirts without stripes" into Google and getting shirts with stripes back: proof that search built for humans behaves like a recommendation engine, not a database that returns a complete, exact result set. - Embeddings over keywords alone. Exa (originally called Metaphor) was built on combining keyword search with embeddings, betting early on what he calls being "bitter lesson-pilled": stack more layers and use neural networks for the complex queries keywords can't handle. - Speed and cost as first-class metrics. A 200 millisecond search endpoint for latency-sensitive systems like voice agents, plus a token-extraction feature that pulls only the roughly 100 most relevant tokens out of a document to cut downstream LLM costs. - Exa Connect. A marketplace where private data providers plug into Exa's API so agents can search proprietary data alongside the public web, with providers setting their own price. - Scale. Exa now serves more than 5,000 companies and 400,000 developers, powering search inside products like Cursor and HubSpot. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Voice conversations break under a constraint text chatbots never feel: a round trip from the user's last word to a reply has to land under a couple of seconds or the conversation stops feeling real, and users change subjects mid-sentence and interrupt constantly. @paularambles, an iOS engineer at Tolan, breaks down how her team holds that constraint in "Tolan: Voice-First AI Companion," on @aiDotEngineer's YouTube. The talk is a full look at building a voice-first AI companion around sub-two-second latency, tiered model routing, and a memory system built as retrieval rather than transcript. - Latency is the product, not a number you check at the end. A drift from two seconds to two and a half tanked nearly every metric, so each stage of the pipeline (end of utterance, transcription, first token, first byte of speech) gets measured separately. - Turn-taking has to read real speech. A short "yes" or "yeah" won't register as a turn and curse words get stripped out; a turn-taking system that reads speech patterns cut early, wrong interruptions by more than half, for about 60 milliseconds of added latency. - A tiered model fleet, routed by stakes, not cost. A cheap classifier called the Tone Router reads the emotional state of every turn; a user's first conversation or anything read as crisis or therapist-style always goes to the frontier model, while lighter turns and background work run on smaller ones. Moving to GPT-5.1 on the Responses API alone cut time to speech by more than seven tenths of a second. - Memory as retrieval, not transcript. Facts, preferences, and emotional signals get embedded into a vector database with sub-50ms lookups, compressed nightly by merging duplicates and resolving contradictions, and split into a volatile live tail versus a stable, cacheable block of what actually gets recalled. - Context gets rebuilt every turn, not reused. A cached context becomes actively wrong the moment a user pivots mid-conversation, so each turn reassembles a summary, persona card, retrieved memories, tone guidance, and app state from scratch. - Claude co-authors more of Tolan's iOS code than any individual engineer. A fleet of implementation, review, PR-shepherd, and bug-triage agents backs it up; crash-free rate rose from 99.6% to 99.9% and runtime errors dropped by more than half. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Voice agents that sound great in a demo tend to fall apart within days of going live, and the four-layer pipeline underneath (speech to text, LLM, TTS, turn detection) is exactly where it goes wrong. @bevenky, founder and CEO of Plivo, walks through where it actually breaks in "5 Voice Agent Failure Modes You'll Hit in Week One," and @aiDotEngineer posted the talk on YouTube. The patterns come from Plivo's own scale: the company sees over a billion voice calls a month on a telephony stack it's run since 2011, and the numbers Venky cites come from what he's seen in production with customers. - Latency is a three-way tradeoff. Frontier models like OpenAI, Claude, and Gemini run a P50 time-to-first-token around 450-500ms but spike to 1.2-1.3s at P90/P95, which is enough to make callers hang up. Open source models like Qwen 3.5 and Gemma 4, self-hosted, can hit under 300ms and are what Plivo actually runs in production. - Transcription is brittle by default. Word error rate runs 4-6% for state of the art engines but hits double digits on real, accented, noisy calls, especially on proper nouns, phone numbers, and addresses. His fix: dynamic keyword boosting instead of static, plus an LLM pass to post-process the raw transcript before it hits your agent logic. - Data collection needs typed fields, not free text. Treating collected values like Pydantic or Zod fields instead of parsing open transcripts took Plivo's collection accuracy from around 30% to 95-97%. He also argues for testing each field as its own unit test rather than running full end-to-end agent evals. - TTS needs a normalization layer of its own. Strip markdown and emoji before synthesis, load custom pronunciation dictionaries for names and brands, and slow playback to around 0.7-0.8x when the agent hits an entity it needs to enunciate. His own test case: whether the agent can say his last name, Balasubramanian, and the company name, Plivo. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

A 1% error rate across a trillion phone calls a year works out to 10 billion incidents. @sumanyu, founder and CEO of Hamming AI, does that math in "1 Trillion Phone Calls/yr, 10% Error rate: The Crisis in Voice AI", then argues the real number is worse: across the 10,000 voice agents his company monitors, the actual error rate runs closer to 10%. @aiDotEngineer has the talk on YouTube. It's a working framework for finding and fixing the failures that don't show up in a demo, plus a look at what adversarial testing turns up once real users get on the line. - What failure actually looks like. Agents that claim they verified eligibility they skipped, apply discounts they weren't supposed to, mishear a request, or say they booked an appointment that never made it onto the calendar, the same failure that sent Sumanyu to a doctor's office with no appointment on file. - A five-step loop for closing the gap. Borrowed from friends who worked in growth at Facebook: identify what's breaking, prioritize by frequency and severity, figure out the fix, ship it, then check for regressions before going back to monitoring. - From gut feel to real coverage. Most teams start by listening to calls manually, then move to spreadsheet rubrics for greetings, closings, and core logic, then to an evals product running LLM-as-judge scoring, but Sumanyu argues the real gains come from analyzing patterns across conversations, not just within one. - Testing beyond replay. Rerunning a real failed call dozens of times is a start, but the talk covers varying wording, accents, and intents to stress-test a fix, plus A/B testing for things that don't simulate well, like the first five seconds of an outbound call. - Red-teaming turns up real breaks. Hamming's red-teaming product, shipped in April, has broken about one in five agents tested across financial services, healthcare, and consumer use cases, bypassing verification steps and pulling data it shouldn't have had access to. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Most voice agents today behave like a walkie-talkie: one side talks, the other waits, and that turn-taking is where the friction lives. @neilzegh, co-founder and CEO of Gradium, takes that on in "Your Voice Agent is Just a Walkie Talkie," on @aiDotEngineer's YouTube. The talk traces how voice interfaces got here and lays out the tradeoff between naturalness and intelligence that every voice agent architecture has to make. - Three generations of voice interfaces. Siri's 2011 launch ran on a closed-ended natural-language-understanding pipeline built for narrow commands. OpenAI's original voice mode swapped that for an LLM and opened up the dialogue, but couldn't take real actions. Today's cascaded STT-LLM-TTS voice agents add tool calling and planning back in, so they can actually complete tasks. - Backchanneling exposes the walkie-talkie problem. A live demo shows a turn-taking speech-to-speech model interrupting every "mmhmm" and "yeah" a listener gives, because the model can only be speaking or listening, never both. - Full duplex needed a new architecture, not just lower latency. Multi-stream language models predict two token sequences at once, one per speaker, so both sides can be active or inactive independently. That's how Moshi, released in 2024, became the first full-duplex speech-to-speech model, and how Hibiki became the first simultaneous speech-to-speech translation system. - Audio has to be compressed before an LLM can touch it. An eight-word, three-second sentence at 24kHz is 72,000 raw waveform timesteps. Since transformer cost scales quadratically with sequence length, that's on the order of 100 million times more expensive to process directly, which is why neural codecs compress audio into dense tokens first. - Naturalness and intelligence trade off against each other. A model's capacity is fixed, so teaching a text LLM to also understand and produce speech spends some of that capacity, and it comes out of intelligence. - Gradium's bet: split the model. A small, natural, full-duplex voice interface handles the conversation and hands reasoning, tool calls, and planning to a background text LLM. The talk frames this as cheaper than routing chit-chat through a giant multimodal model, and as a way to keep the freedom to swap text backends, noting that OpenAI's advanced voice mode stayed on GPT 4.0 long after newer text models existed. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Speech doesn't have to be the response a voice model gives back. @charlierguo, a Developer Experience Engineer at OpenAI, builds "Voice Agents Can Just Do Things" around that idea, and @aiDotEngineer has it posted on YouTube. The talk lays out a framework for deciding when a voice agent should actually talk, act, or just wait. - Three modes, not one. He frames voice design as speech-to-speech, speech-to-action (the model uses tools), and event-to-speech (the model reacts to something and speaks), and argues the best products mix all three rather than picking one. - Speech-to-action is the underused one. Form filling, creative tools, and computer use (he points to the ghost-cursor animation that clicks around the screen for you) are all cases of voice acting on your behalf instead of just replying. - Event-to-speech is still exploratory. Hands-free moments, like cooking, and proactive notifications that escalate from a visual cue up to actual speech, are the two patterns he points to here. - Native audio over the chained pipeline. OpenAI's Realtime model family skips transcription and works on audio tokens directly, since, as he notes, only a small fraction of face-to-face communication is carried in the words themselves. - GPT Realtime 2. The newest Realtime model reasons before it speaks, can delegate parallel tool calls, and uses preambles so it can tell you it's about to go do something, the way a travel agent might say they need a couple seconds to check flight prices. - Accessibility as the throughline. He points to developers who lost hand mobility and dexterity, and who now generate far more code with the help of voice and coding agents. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Cascaded voice agents borrow their shape from self-driving cars: perception, planning, and controls. @bobowchan, a staff software engineer at EliseAI, walks through that architecture in "Realtime Voice Agents with Frontier Intelligence," on @aiDotEngineer's YouTube. The talk lays out the harness tricks that let a voice agent stay fast even when the language model doing the thinking is slow. - A transcriber with two speeds. The "streaming speculative transcriber" layers a fast streaming model, Flux, under a slower but more accurate one, Scribe v2. The fast layer gives instant guesses; the accurate layer corrects them once it has enough context, like catching that a garbled string is actually a name. - Tool calls that don't cost a round trip. A background agent handles tool calling and writes the result back into the main agent's context, so the main agent behaves as if it made the call itself without waiting on it. - Generation that bets and cancels. Each new transcription update triggers an early attempt at the agent's response. If the next update changes the picture, that attempt gets thrown out and regenerated, rather than waiting for the user to fully finish before starting any work. - A prefix cache for speech. Before playing audio, the system checks whether it's already generated speech for that sequence of words, waiting for three words before it will use a cached hit. Common phrases like "you said your name is" play from cache while Cartesia's WebSocket stream keeps generating the rest, and Cartesia's own audio for the cached portion gets muted out. - A recorded call as proof. Bo plays back a real call, an OBGYN office booking an ultrasound, to show the transcriber, tool calling, and speech layers working together in one conversation. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

AGI will not be typed, it will be spoken: that's the closing line of "Speech-to-Speech Model Research at Google DeepMind," the talk @valeriawu_ and Tom Ouyang, both of Google DeepMind, gave, and @aiDotEngineer has posted it on YouTube. It traces how Gemini's natively multimodal pretraining replaced the old pipeline of separate acoustic, pronunciation, and language models, and shows what one speech-to-speech model can do once translation, tool use, and conversation all run through it. - Live translation. Gemini's live model does real-time speech translation across 70+ languages, preserving each speaker's voice, with streaming quality that holds up against offline systems that get to hear the whole utterance first. - Three things pulling against each other. The talk frames the model's goals as conversational quality, intelligence, and multimodality: push "thinking" higher and evals show more intelligence, but latency and naturalness take the hit. - Knowing when to stay quiet. A feature called proactive audio lets the model decide not to respond when it hears background noise or someone else talking, since most real conversations aren't happening in a quiet room. - Non-English first. Most Gemini users aren't English speakers, so translation and localization work covers all their languages, not just English. - Two very different demos, one model. A Search Live clip identifies a boucle sofa in Spanish, localized to Spain's Spanish, correctly leaving "mid-century" in English; a Live API demo has the same model handling a roadside assistance call, reading back a car's registration plate and postcode. - Faces, not just voices. A pilot with Citi shown at Cloud Next adds real-time, multilingual, lip-synced avatars on top of the same model. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Most people never get to test their morals against a hard case until they're actually living it. @brendanh0gan, a Machine Learning Scientist at Morgan Stanley, built a way to do it ahead of time: an open source game where you write down your morals, one agent codifies them into a legal system, and two adversarial agents hunt for contradictions in it. @aiDotEngineer published the talk, "Loophole: Adversarial Agents To Stress Test Your Morality," and it's on YouTube. The talk walks through how the adversarial loop actually works, then traces three directions Brendan has taken the project since. - Where the idea came from. After opting out of DNA forensics use with 23andMe, Brendan started thinking about legal systems as attempts to codify moral beliefs, and how case law exists precisely because that translation is hard to get right in general terms. - How the game runs. One agent drafts your morals into a formal legal system with articles and sections. One adversarial agent looks for loopholes, things that are immoral but legal under your system. Another looks for overreach, things that are moral but illegal. A judge agent either auto-patches the code or raises the contradiction to you. - Constitutions for chatbots. The same process can draft a system prompt with what an agent should and shouldn't discuss, then run adversarial agents against it to find breaks, an approach Brendan compares to GEPA. - Decentralized contracts. Codify how you want your data or your work handled, then run that against a company's terms of service or another party's contract to surface disagreements before you sign anything. - Simulating the US Senate. Brendan had Claude build a legal code for each senator from their voting history, then simulate votes on real bills and hill-climb the bill's language. One Medicare bill went from a 50-50 split to passing 52 votes by adjusting wording without violating the bill's core tenets. - Simulating constituents. Using NVIDIA's dataset of US personas, 500 per state, the same process builds a legal code per persona so a bill's language can be tested and optimized against a broader slice of public opinion. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!