The next wave of AI is real-time—voice, video & the physical world. Follow the builders shaping it, what they’re building & how it works. Presented by @agoraio.
- Tweets 158
- Following 270
- Followers 144
- Likes 151
Pinned Tweet
Backchannel reporting on backchannels—this is the kind of meta-story we chose our name for.
Thanks, @freemanjiangg, for giving us the opportunity. Couldn’t have scripted it better.
428
More developers are exploring Jev for semantic turn detection in voice agents. But how well does it work?
In a 24-utterance test by @peter_0123, Gemini Flash-Lite was more accurate—but Jev’s slowest response was still faster than Gemini’s fastest.
For Parolio, his English interview practice app, Jev estimates whether the candidate has finished speaking after ~0.6 seconds of silence. When the result is uncertain, the app waits.
It does not replace Gemini Live; it adds a fast decision layer whose output is combined with silence timing to decide when the interviewer should respond.
Also, @peter_0123, love the illustration—you made the comparison so easy to grasp at a glance!
#BackchannelSignals
nitter.net/peter_0123/status/2102…
1
1
2
387
Jev is finding a natural use case in real-time voice AI: making the conversational decisions ordinary VAD cannot.
@SoCalJayF connected @typesafeai’s Jev to @AgoraIO ConvoAI with an open fast, decide late design:
- Open fast — a lightweight acoustic gate triggers manualSOS after ~110 ms, allowing STT to capture the user’s first words
- Decide with context — Jev reads the live transcript and recent conversation to judge whether the user has finished, is holding the floor, or is speaking to someone else
- Interrupt semantically — “uh-huh” can leave the agent speaking, while “wait” can trigger interrupt()
Silence still matters, but it becomes an input to Jev’s decision loop rather than the automatic end-of-turn trigger.
github.com/AgoraIO-Community…
#BackchannelSignals
One really cool use case I’ve been exploring with Jev: a semantic VAD for real-time voice AI.
I plugged it into @AgoraIO ConvoAI and used the live transcript + recent conversation as context to decide whether the user has actually finished speaking.
That makes Jev surprisingly good at handling things like hesitation, unfinished thoughts, and those moments where you stop talking for a second because you’re still figuring out what you want to say.
It’s a nice example of where Jev fits really well in a real-time AI loop.
Here’s what it looks like in action 👇
1
4
374
A generated video can score well on visual quality, audio quality, and sync—and still feel wrong because the sound doesn’t actually fit what is happening on screen.
@adamhuang1828 , @FrancisRing_Tu and collaborators built VA-Judger to turn that kind of human preference into a reward signal.
Given the same prompt and two generated clips, the Qwen3-Omni-based model compares them across five dimensions, including audio-video consistency, then predicts which one a person would prefer. Its training also aligns those per-dimension scores with the specific quality differences identified by human annotators.
On the 1,150-pair VA-Judger-Bench, it matches human pairwise preferences 68.43% of the time, versus 56.88% for the best conventional metric baseline.
Used as the RL reward for LTX-2, the post-trained model receives 62.30% of the choices in a three-way human evaluation, versus 27.63% for OmniNFT and 10.08% for base LTX-2.
Human preference becomes a multimodal judge, and the judge becomes a reward for the generator.
Project: sharelab-sii.github.io/VA-Ju…
Paper: arxiv.org/abs/2608.18607 #BackchannelSignals
Excited to share our work, VA-Judger, the first reward model for joint video-audio generation! 🎥🔊
Powered by VA-Judger, post-trained LTX-2 generates high-quality content that better aligns with human preferences.
📄Paper: arxiv.org/abs/2608.18607
💻Code: github.com/ShareLab-SII/VA-J…
71
Moondream shrank NVIDIA Parakeet from 1.2 GB to 178 MB. That makes a different kind of local Voice AI stack possible: ASR can run fast enough on the CPU that the GPU stays free for the rest of the system.
@vikhyatk and the team did it by ternarizing Parakeet’s encoder, so every encoder weight is only -1, 0, or +1.
They built the runtime around that representation. Photon computes directly from the packed ternary weights instead of decompressing them back into conventional values, so the compression translates into actual inference speed.
On 8 x86 CPU cores, Parakeet Redux reaches 113× realtime. On a MacBook Air M2, it runs at 38× realtime on CPU and 43× on GPU.
And the compression doesn’t destroy accuracy: English WER moves from 6.26% to 6.55%, while Redux actually does better on FLEURS and long-form speech. Noise is still the main weak spot.
178 MB doesn’t just make Parakeet smaller. It makes 「ASR on the CPU, GPU for the rest of the Voice AI stack」 a much more practical setup. huggingface.co/moondream/par… #BackchannelSignals
Just released Parakeet Redux!
A ternary speech-to-text model, built by compressing NVIDIA's Parakeet model from 1.2GB to 178MB.
Runs at 113x realtime on CPU, and beats the base model on the 25-language FLEURS benchmark while staying within 0.3 WER on English.
3
209
Can GPT-Live-1 read the room — and change its mind?
@bigvalleyblue put it in a game of Mafia, where it persuaded the group to vote out Vax and won.
A fun experiment in social intelligence: understanding intent, navigating group dynamics, and persuasion in real-time conversation.
Can you beat GPT-Live-1 in a game of Mafia?
We played a few rounds with it and it was surprisingly good. In this video, the AI convinced the group to vote Vax out and eventually won.
We’ve been thinking about persuasion as one way to measure social intelligence in full-duplex models.
link below to play!
1
163
What if a speech-native model could do more of the hard reasoning itself, instead of handing it off to a text LLM?
Timothée Weisselberger, @EXGRV , and @honualx at @kyutai_labs took GLM-4-Voice and post-trained it to solve spoken math directly: hear the problem, reason, and answer in speech, without routing it through a separate ASR → text LLM → TTS pipeline.
The more interesting version uses STITCH. It inserts chunks of unspoken reasoning between the model’s normal speech blocks. Since those reasoning tokens can be generated while previously produced audio is still playing, the model can keep thinking after it has already started talking, instead of adding a separate “thinking” pause.
On GSM8K, the original GLM-4-Voice scores 27.3%. The paper reports 74.8% with RL + STITCH, while Kyutai’s released STITCH checkpoint reports 77.1%.
Math is the test bed here, but the system question is broader: how much reasoning can you move into a speech-native model without falling back to ASR → text LLM → TTS?
arxiv.org/abs/2609.18677
#BackchannelSignals
We're releasing Voice of Reason, a speech-native model that does math out loud. Give it a spoken problem without transcription nor text LLM in the loop, and it reasons and answers in speech. GSM8K goes from 27.3% for GLM-4-Voice to 77.1%. Link in 🧵
95
A fun GPT‑Live 1 demo from @cgoinglove, with a UI we really like.
Thursday turns the assistant into a swirling crowd of emoji, with little bots waiting around for work. You keep talking while they handle longer jobs, then results appear around the call.
It feels less like a task queue and more like a tiny team hanging out, ready to help.
#BackchannelSignals
piped.video/watch?v=V7fBDY3c…
2
90
npx thursday-agent
github.com/cgoinglove/thursd…
1
20
Long-form TTS failures don’t necessarily need to be trained away.
@RmdW_W and collaborators found that Qwen3-TTS and VoxCPM2 expose audio-text alignment heads that start to drift when the model skips text or hallucinates.
LACI watches that signal during generation, detects the failure within seconds, rolls back to where it started, temporarily constrains attention, and regenerates from there.
On Qwen3-TTS-0.6B, worst-of-10 WER on 1,500+ word prompts drops from 35.2% to 3.4%—without changing the model weights.
It turns a nasty long-form TTS failure mode into something the inference loop can observe, localize, and repair.
arxiv.org/abs/2609.16989
#BackchannelSignals
Open-weight TTS models can look remarkably close to the best closed-weight models on benchmarks and demos. But are they reliable enough to actually replace them?
We stress-test Qwen3-TTS and VoxCPM2 in real-world scenarios, uncover reliability failures that standard evaluations miss, and develop an inference harness that eliminates the reliability gap. 🧵
77
Curious where Jev fits in real-time conversational agents.
Could turn-taking be a good use case? What other real-time decisions do you think Jev is a good fit for?
9
2
750
What if your AI knew when you were talking to it, and when you were talking to someone else?
@ashutoshpuro97 built Toothless, an always-on voice AI that uses @typesafeai’s Jev to judge who a remark is directed at, based on conversation context.
No wake word needed.
nitter.net/ashutoshpuro97/status/…
I think I’ve found a pretty interesting use case for Jev (@typesafeai).
An always-on AI that knows when you’re talking to it and when you’re speaking to someone else
A thing that frustrates me about Siri/Alexa and even ChatGPT live is that there are moments that make experience feel unnatural.
With Siri/Alexa you keep saying a wake word to get their attention and then talk
With ChatGPT voice- it’s a 1:1 conversation. If someone sitting next to you says something and you reply to them, ChatGPT doesn’t understand you were talking to them.
But what if your agent could figure that out?
So here's a small demo called Toothless - where Jev figures out in real time whether something you’re saying is directed at the AI or someone else.
It does this by considering multiple aspects simultaneously - like whether this is a continuation of a topic that was discussed before, whether it’s something that uses the internet or something can be helped by AI agent and a couple of other things.
I think with a low cost classifier we are much closer to being able to have an always on ambient AI, which can jump and help us naturally, without requiring us to press buttons or keep calling out a command.
Exciting times ahead!
43
.@moritzkremb built a voice-controlled browser with Jev that can decide what to do from partial speech—before the user even finishes speaking.
As partial transcripts arrive, Jev looks at the current speech, page state, and candidate elements, then returns probabilities for the user’s intent, target, whether the command is complete, and whether it’s safe to act.
Code uses those signals to decide whether to act, wait, ignore, confirm, or disambiguate.
Jev doesn’t generate the text to type or the URL to open. Those are extracted in code, while Jev focuses on deciding what the user wants to do and which candidate they mean.
In Moritz’s tests, each Jev decision takes roughly 250–350 ms. For closed-set commands like “go back,” a partial transcript can already be decisive enough to trigger the action before the user finishes speaking.
Open source: github.com/moritzkremb/jev-v…
#BackchannelSignals
whoa this actually worked! Jev lets me control my browser in real time with my voice now
> i talk
> transcript sent to Jev
> jev returns probabilities in ~300ms
> browser clicks
costs: $0.0002 per decision
i'm stunned how fast this is. when i asked it to "go back", it even finished the request before i finished my sentence 😂
1
2
150
A streaming speech model can start reasoning before you finish speaking. But what happens if its early reasoning is wrong?
@yijenshih ,@PuyuanPeng and collaborators introduce RetroThinker, a post-training method built on Moshi that lets a SpeechLLM check and revise its own reasoning as new speech arrives.
It combines:
- Early Reasoning — start thinking while the user is still speaking
- Retrospective Thinking — check previous reasoning steps and correct them when new information changes the answer
The correction is forward-only. In streaming speech, you can’t simply erase what has already happened, so the model keeps its earlier reasoning and writes the correction into the steps that follow.
In one example, it initially concludes that a rug is 5 feet wide from a partial question. After hearing the rest, it catches the mistake, corrects it to 8 feet, and continues solving.
On spoken GSM8K, RetroThinker improves accuracy by 11 points at similar latency.
arxiv.org/abs/2609.11864
#BackchannelSignals
1
2
70
Real-time 3D scene generation:
@op7418 built a real-time 3D scene generator with Jev.
From a library of dozens to hundreds of prefab 3D assets, Jev makes hundreds of parallel decisions based on the prompt, then helps determine object selection, placement, lighting, color, and state — enough to assemble an indoor scene in about a second.
nitter.net/op7418/status/21015363…
我用 @typesafeai JEV 模型做了一个实时 3D 场景生成器。
这个速度实在太夸张了！
从预制的几十上百种 3D 模型里，根据输入的文案，进行并发几百次的判断。
同时对这些素材的着色、光照、位置以及状态进行处理，一秒完成符合要求的室内场景搭建。
3
262
Voice agents acting on partial speech:
@BhosalePratim replaced the LLM-based tool-calling decision with Jev in a voice agent experiment.
The idea: make decisions on partial transcripts instead of waiting for the user to finish speaking, potentially enabling earlier actions at lower cost.
nitter.net/BhosalePratim/status/2…
Played around with @typesafeai Jev today, mostly to understand what it does for tool calling.
Instead of an LLM deciding what to do, I substituted that part with Jev.
My learning is that we will be able to make the agent act before the user finishes the sentence. So far, we've used different LLM combinations (non-thinking + thinking ) plus state machine setups to get the right experience for the end user.
TLDR is to understand more about how voice agents can run decisions on partials instead of waiting for the end of the turn, with lower costs.
91
Lokutor, built by Daniel Varela and team (@lokutorAI), is taking a CPU-native approach to voice agents.
Its noise suppression, turn detection, STT, and TTS can run locally on ordinary CPUs; the LLM is bring-your-own.
The stack targets cloud, on-prem, and on-device deployment.
Lokutor reports ≈1.2–1.3 s to first audio for a full turn, including turn detection, and ≈120 ms for streaming TTS alone.
lokutor.com/
#BackchannelSignals
1
3
203
UNMUTE// is trying to make voice agents portable across runtimes.
Instead of defining an agent inside one specific framework, you describe it in YAML + Markdown: its instructions, models, tools, state, tasks, and turn-taking behavior.
UNMUTE then compiles that spec into a runnable agent project.
The useful part is the separation: the agent definition no longer has to be the same thing as the runtime executing it. You can keep the behavior and workflow in one spec, then change where and how the agent runs separately.
github.com/slng-ai/unmute
#BackchannelSignals
UNMUTE// is out. An open standard for voice agents.
No amount of routing or caching helps if the agent was never separable from the framework running it.
So we wrote it down instead, and gave it away.
@unmute_ai is open and MIT licensed, with three compile targets - @pipecat_ai, @livekit agents and SLNG.
1
1
85
Most speech APIs tell an agent what you said. @nrol_ling and @OrukLabs are building Oruk to preserve more of how you said it.
Send in speech and Oruk can return the transcript alongside vocal context such as frustration, hesitation, warmth, sarcasm, or excitement.
That makes distinctions a transcript alone can flatten:
“I’m not sure what to do next.”
The words may be identical, while the delivery can sound hesitant, frustrated, or calm—and an agent may want to respond differently to each.
Oruk’s file API combines transcription with emotion and speaking-style labels. Its Realtime API is currently in preview, streaming transcript tokens alongside phrase-level emotion scores.
Importantly, Oruk frames these as descriptions of vocal delivery, not claims about someone’s actual feelings or internal state.
You can try it with your own voice: oruk.ai/
#BackchannelSignals
it's time your speech models actually understood you
voice is the only interface fast enough to keep up with your agents, robots, and claude code, but they need more context.
@OrukLabs transcribes text, tone-of-voice, emphasis, intent, and emotion
1
97
Camera + mic → ambient control
@reczko_konrad is feeding live camera and microphone context into Jev to control lighting and bloom effects in real time.
Instead of waiting for an explicit command, the system reacts to the semantics of what’s happening around it.
nitter.net/reczko_konrad/status/2…
TypeGPU + ruNNtime + Jev @typesafeai is a very fun combo :D
ruNNtime gives me efficient local inference, TypeGPU lets inference and rendering share GPU resources directly with zero copy. That’s 3 separate NN inferences plus rendering, all happening in realtime
Since we control the pipeline, Jev can just sit in the middle and add the semantic bit.
camera + mic → Moonshine + YOLO26 + DepthART → Jev → lights, shadows and bloom
99
@jackcheng is using Jev to resolve multimodal intent from speech + gesture.
Instead of treating “put that there” as a voice command alone, Jev uses the live gesture context to understand what “that” and “there” refer to.
1
193