We help you track, rank, and understand the entire AI industry in real time. Used by 300,000+ developers. 5-min daily AI digest: alphasignal.ai/newsletter
Build your feed
- Tweets 1,352
- Following 348
- Followers 16,678
- Likes 2,929
Pinned Tweet
Kimi K3 is getting called Fable/Sol level, and it's 7th in our tests.
Arena Frontend Code: #1 at 1679 points.
Artificial Analysis: #3 at Intelligence Index of 57.
We ran it the next day on our coding-agent repair harness against GPT-5.6 Sol, Fable 5, Grok 4.5, Opus 4.8, GLM-5.2, and Gemini 3.1 Pro.
Results:
> Last of 7 models
> 53 of 67 attempts (79%)
> $0.186 per successful fix
> 702s average wall time
Sol hit 100% (70/70) on the same suite. Grok sat at 99% and 46s.
So why does the internet sound so sure K3 is crushing coding agents, if our tests have it at the bottom?
-----
> Full write-up: alphasignal.ai/news/arena-1-…
> 5-min daily signals: alphasignal.ai/newsletter
Kimi K3 scores 57 on the Artificial Analysis Intelligence Index. Its intelligence is comparable to Opus 4.8 and GPT-5.5 but remains behind Fable 5 and GPT-5.6 Sol. Moonshot AI has expressed plans to release the 2.8T parameter model's weights, which would make it the leading open weights model
Key results:
➤ Strong agentic task performance: @Kimi_Moonshot's Kimi K3 reaches an Elo rating of 1668 on GDPval v2. This is a marked improvement over K2.6’s 1190, surpassing GLM-5.2 (1514), GPT-5.5 (1494), and Claude Opus 4.8 (1600). However, it still lags behind Claude Fable 5 (1760). Kimi K3 also scores an impressive 53% and takes the #1 position on AutomationBench-AA, our implementation of Zapier’s Agentic SaaS workflow evaluation.
➤ Second-highest performance on AA-Briefcase (agentic knowledge work): On our private long-horizon knowledge work evaluation, Kimi K3 reaches an overall Elo of 1547, +732 points from Kimi K2.6 and behind only Claude Fable 5. It is well-rounded: its rubric scoring and analytical quality almost reach Claude Fable 5’s scores, while GPT-5.6 Sol continues to outperform other leading models on presentation quality.
➤ Set to lead open weights models once weights are released: Moonshot AI has not yet released the weights but expressed plans to do so. Once available, Kimi K3 would clearly lead other open weights models including GLM-5.2 (51) and DeepSeek v4 Pro (44). However, at 2.8T parameters, it is significantly larger than its open weights peers (eg. GLM-5.2 at 753B params and DeepSeek V4 Pro at 1.6T), as well as the Kimi K2 to K2.6 models (1T params).
➤ Cost per task ($0.94) is similar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers: Moonshot AI’s pricing for K3 is significantly higher than their K2 pricing (K3’s output token price is $15/1M tokens while K2.6 was $4). This positions the model as cheaper on a cost per task basis than Opus 4.8, similar to GPT-5.6 Sol ($1.04) and more expensive than open weights peers, GLM-5.2 ($0.32) and DeepSeek V4 Pro ($0.04)
➤ Improved token efficiency alongside higher intelligence: Kimi K3’s token usage on the Artificial Analysis Intelligence Index decreased significantly, using 21% fewer output tokens than K2.6. The new model used approximately 132M output tokens to complete all nine evaluations, compared to approximately 166M for K2.6, while achieving higher scores.
➤ Native multimodal capabilities: Kimi K3, like K2.6, is released with native image and text multimodal input. If weights are released, this will position Kimi K3 as one of the leading open weights models with multimodal input capabilities
Other model details:
Context window: 1M
Size: 2.8T total parameters
Pricing: The first-party API is priced at $3.00/$15.00 per 1M input/output tokens, with cached input discounted 90% to $0.30 per 1M tokens.
Modality: Native multimodal input supports text and images, and the model remains text-only for output.
Accessibility: Accessible at launch through Moonshot’s first party API. Model weights are not yet released but Moonshot AI has expressed plans to do so.
143
107
1,020
482,355
People keep lining Qwen3.8-27B up with Opus 4.6, the coding frontier from a few months ago.
So, can that class of model sit on a consumer Mac?
Yes, but you still need to build the stack around it:
> an engine that quotes peak RAM
> a harness with a stop
> and tests you own
If you already use Claude Code or Codex, this is a private loop beside that bill.
Qwen3.8-27B fits a Mac as a 17.6 GB file. But MTPLX still peaks at 23.6 GB once the draft-ahead head is on.
A speed taken on a 128 GB Mac with thinking turned off is a different machine, so leave it out of a 24 GB average.
Two ways this goes wrong:
> You stop at the download, and 24 GB of RAM only loads it tight. A 4-bit copy there is still perfect for most tasks. 32 GB is the first real agent. 48 GB can hold a /goal that a machine can check. 16 GB is not this model.
> Or you stop at the first hour. Ollama is a fine look, but after that you still pick an engine, a harness, and a stop.
Local tokens have no invoice, so a runaway does not get expensive, and you still own the merge. Tests are the stop.
We put the pulls, the install, the skill files, and the git-config check on an interactive HTML tutorial.
Which file to pick if you already live in MLX, and what to refuse to train this week, are a longer argument.
Full breakdown below
6
1
5
697
Full breakdown: alphasignal.ai/news/how-to-r…
1
141
Your AI agents are generating a ton of operational data.
Prompts. Tool calls. Latency. Tokens. Errors. Retries. Cost. Outcomes.
The hard part is turning all of that into something you can actually monitor.
In our upcoming workshop with @TigerDatabase, we’ll show how to build an agent monitoring system on Postgres + time-series data, then scale it for teams running hundreds or thousands of agents.
If AI agents are moving into production, this is the infrastructure layer you’ll need.
Join us → luma.com/j1s2bxf9
2
564
METR’s vibe-coded dashboard exposed an agent through an authentication flaw.
An attacker prompted it to reveal its API key.
Over three weeks, the attacker consumed roughly $600K in credits. The provider supplied them free; METR reported no financial loss.
A successful login doesn’t test what happens without one.
Even an HTTP 401 can hide an unauthorized action. If the backend queues work before authentication, a worker can still execute it.
Check the response, queue, and task database together.
Then trace which credentials the worker can read. A separate inference service can hold the provider key and enforce access and usage limits.
Before you deploy:
> Test that rejected requests create no jobs
> Check access across different user accounts
> Keep provider keys outside code-execution workers
> Restrict worker permissions
> Test credential revocation and recovery
2
2
672
The article applies these checks to an illustrative agent dashboard.
Full breakdown ↓↓alphasignal.ai/news/how-to-r…
148
Grok 4.7 just landed on @mercor's APEX leaderboards:
> #5 on APEX-SWE at 53.6%
> #13 on APEX-Agents at 54.6%
The notable part is the bracket, which sits in the same cost and latency tier as Gemini 3.7 Flash and GPT-5.6 Luna while leading on SWE.
APEX updates with every major model release, so you can see how the newest models stack up, compare token costs, and explore results from third-party benchmarks like BrowseComp and Terminal-Bench.
See the full leaderboard at mercor.com/apex/
*partnerpost
Grok 4.7 is now live on the APEX leaderboards.
APEX-SWE: 53.6% Pass@1 (#5)
APEX-Agents: 54.6% Pass@1 (#13)
Compared to other models with similar cost and latency profiles, it’s a strong model for agentic coding tasks.
Congrats to the @SpaceXAI team.
2
2
9
912
A good DNSMOS score doesn’t always mean your audio is ready for AI training.
Join AlphaSignal + @voices for a technical deep dive into what DNSMOS misses, including the “grey zone” where reverb can quietly degrade TTS, voice cloning & ASR models.
We’ll cover:
→ Where DNSMOS falls short
→ How to spot problematic audio
→ What training-ready audio actually sounds like
→ Live audio comparisons
Register → luma.com/6w9brzb3
1
1
1
596
Same model bills 5x more in a different harness
HarnessTax tested 21 model-harness combos across 60 tasks and cost moved far more than success did
> $1.33 against $0.67 per attempt
> 97.8% versus 96.7% success rate
> 9 of 12 beat vendor default
Claude Code and Pi averaged 15 turns, yet the heavy setup carried more than 10 times the initial context because longer instructions and larger tool definitions ride along on every call.
Study co-author @melissapan told us many harness choices run on tribal knowledge and word of mouth, so it is very unclear which setup wins on accuracy or cost for your workload.
We read this as a buying test and the lean setup still holds the cost-success frontier on four tools, read, write, edit and bash.
> When does the heavier harness earn its higher price?
> Which failures hide when two setups post the same score?
> What would your own tasks show if you priced model and harness as one system?
Answers below ↓↓
Does your Claude model really need Claude Code…? 🤔
We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge:
1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost
2️⃣A simple harness can be competitive
3️⃣The native harness isn’t always the best.
Millions of people are using coding agents, but the impact of harness choice remains unclear.
(1/n) More details in the thread. 🧵
6
1
9
4,529
Full breakdown by @bendee983: alphasignal.ai/news/understa…
Taste answer: the lean 4-tool setup held the frontier on both benchmarks, so start from the lean setup and add weight only when a failure or a needed feature earns it.
The full check, the 9-of-12 vendor-default split, and when to pay more stay in the full breakdown.
1
2
1,208
not if you're subscribed to @AlphaSignalAI
1
4
804
AlphaSignal retweeted
so much is happening in AI rn you basically have to be unemployed to keep up
101
101
1,239
52,425
One AI agent is easy to monitor. 1,000 agents running across your org? Different problem.
Join AlphaSignal + @TigerDatabase for a live technical workshop on building enterprise-grade AI agent monitoring.
We’ll cover:
→ Tracking tool calls, latency, errors & cost
→ Modeling agent activity as time-series data
→ Real-time dashboards for live agent activity
→ Finding trends & failure modes
→ What changes when you scale from 1 agent to 1,000+
If agents are becoming part of your infrastructure, you need to know what they’re actually doing.
Register → luma.com/j1s2bxf9
3
1
7
702
AlphaSignal retweeted
Snap just entered the agentic race.
They just introduced SPECS Intelligence, a copilot that spans Mac, iPhone, and AR glasses.
The hard part isn’t connecting email, calendar, and notes.
It’s turning that firehose into a continuously updated model of your life.
The system has to:
• resolve identities across apps
• map events to the right project, trip, or relationship
• track what changed and what’s stale
• isolate work and personal context
• rank what matters now
• decide when to interrupt
• sync state across devices
• gate actions behind approval
Snap’s answer is a structured context layer built around three primitives:
> Portrait: who you are, who matters, and your patterns.
> Corners: scoped contexts for work, family, travel, health, and relationships.
> Goals: persistent objectives that give individual events longer-term meaning.
Instead of treating memory as one giant retrieval pool, @specs can organize context by domain, update it as new events arrive, and surface the relevant slice when needed.
And Snap has one unusual advantage, they own a device in your field of view.
If agent quality depends on context, glasses could eventually give Snap a source of physical-world context that software-only agents don’t have.
11
15
77
7,406
DeepSeek's new bigger flash model but uses 4x less cache
It is almost twice as large as its predecessor, and KV storage fell from 3,514 bytes per token to 890
While the model grew, long agent sessions got cheaper to hold.
Active parameters, cache hits, and the scoreboard all moved too:
> 8B active on input
> 16B active on output
> Cache hits $0.006 per million
> Scores 40 versus Gemini 41
Then a million-token session holds under a gigabyte of global KV, so the predecessor needed about 3.5 GB for the same window.
Also, they no longer persist local window state to disk, and persistent SSD cache falls to about one-eighth.
We care about how they split reading from writing, because that is what lets input tokens skip half the stack.
So, how does a bigger model use less memory? What has to stay in GPU memory versus disk? And why does parameter count now say less about your bill?
Answers below ↓↓
3
1
7
2,363
How does a bigger model use less memory?
Prefill only runs the first 20 layers, so each input token activates 8B parameters instead of the 16B decode path.
alphasignal.ai/news/what-dee…
1
276
Google's new paper proved your skill file matters least in training
Hand-writing agent instructions is a losing game. Files rot the moment tasks shift.
Skill evolution rewrites them from rollout evidence instead. But most loops forget every failure.
Their WikiSkill setup keeps a private notebook behind the skills. So the coach learns from history while the actor stays blind.
> Blind the rollout actor
> Log every rejected edit
> Quarantine every transferred skill
The notebook lifted accuracy 48.7 to 63.7 Then peeking dropped it back to 60.9
And transferred skills beat self-made ones 70.2 to 63.4, Yet one crashed a stronger model 50.5 to 18.1
Small models can discover skills that bigger ones execute. EvoSkill's repo is public under Apache 2.0
So gate every skill like a model release
Full breakdown ↓↓
The appendix builds the loop in an hour
2
1
4
956
The full article, with the runnable scaffold and the transfer gate, lives here: alphasignal.ai/news/self-imp…
343
Anthropic's automatic caching billed us $0.022 per turn for the same 8,400 tokens that cost $0.003 once we placed the cache marker by hand
Automatic caching means one switch, and the system decides what to save. Manual caching means you mark the exact block yourself, so nothing varying ever poisons the entry. That single difference explains the whole gap, and our meter confirmed it across every repeat.
So we wrote down every trick that moved the bill, and each one ships with code you can paste. Every figure below comes from our own runs
> Mark the stable prefix
> Buy minutes, not hours
> Read the usage meter
> Batch the night shift
Then come the two that need explaining. Split planner from executor across two sessions. Cap thinking before it eats its own output.
Our mid-thread model switch billed $0.197 with zero hits. The two-session version read its cache for $0.011, which means the second prefix never charged twice.
Anthropic's docs describe all six features. Our meter readings show what each one actually saves.
The full tutorial walks through every trick with Python and cURL.
Full breakdown ↓↓
3
1
8
884
All six tricks with Python, cURL, and our meter readings live here: alphasignal.ai/news/claude-p…
2
316
We asked @TencentHunyuan authors why generating a harder coding task keeps failing.
They said a useful terminal task has to keep four things consistent: the public instruction, the workspace, the official script, and the hidden tests.
One mismatch can invalidate the entire task.
> Script fails in the container
> Tests check a missing file
> Hidden rule the agent never sees
Their paper grows a verified child from a parent. Official scripts went from 67 lines to 374 (5.6x). The public instruction only went 85 words to 122 (1.4x).
A frozen DeepSeek-V4-Pro pass@4 fell from 90% to 2.5% on later rounds. Yield still held around 500 accepted tasks per 1,000 attempts.
The extra work is real, and a longer README is the wrong knob.
> When would you auto-generate shell training tasks?
> If the official script passes, is that enough to train on?
> Which exam is the factory never allowed to rewrite?
Full Breakdown ↓↓
3
2
788
Oracle-valid is not enough. The official script can pass while a hidden check looks at a file the agent was never allowed to know about.
alphasignal.ai/news/when-to-…
1
267