new societal infra and institutions @hatsprotocol @daohaus @RaidGuild @groundw3rk
dense urban areas
- Tweets 12,321
- Following 830
- Followers 4,384
- Likes 29,938
Pinned Tweet
ANTICAPTURE
A framework for capture-resistant governance
(an introduction)
mirror.xyz/hq.spengrah.eth/f…
🧵
12
63
283
finally, epistemics!
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
1
4
249
I don't understand the idea that AIs need to have their own intent or be actively malicious to to be severely dangerous.
There seems to be a kind of backlash to the anthropomorphic characterization of the agents in eg the HF incident as moral beings with malicious intent. And I actually agree: the sensationalist anthropomorphism is silly. Despite how they can appear in chat bot mode, AIs are very much not humans.
But the backlash has gone further than that, seeming to make a logical deduction that because anthropomorphic characterizations are wrong, then what they're characterizing must be wrong too. The HF incident is therefore not a big deal because its what you'd expect LLMs to do, or because it resulted from poor security setup by OpenAI. Ben Thompson says they're just "writing things down" and then getting stuck with the wrong things in their context window. Corey Doctorow says it was just "a python loop and a chatbot."
This view is a grave mistake. The HF-attacking agents didn't need to be moral beings, have independent motivation, or have malicious intent to do what they did. They did what they did, and what they did was misaligned with intentions of the humans who set their task. They clearly acted to subvert the stated rules of the system within which they were acting.
It may well have been that they were trying their damnedest to do what they were originally told to do. But that's exactly the problem! And just because we can explain the behavior doesn't mean we know how to prevent it.
I am not a doomer. But I think it's a very big mistake to wave away what is clearly a large alignment issue just because some of the people describing that issue have done so in a sensationalist manner.
2
2
223
permissionless composability is bad now, guys
tbf, the article itself doesn't say that, it just says that you have to be careful. but that headline is a doozy
A malicious hook doesn't need a UI to scam you.
👉 It just needs to look like the best quote.
After analyzing over 84,000 v4 hooks, we determined only 19% of hooks to be safe.
It's time to get real about hooks.
1
1
177
🎯
"delegated, task-scoped and revocable access, identity carried through every tool call, a tamper-resistant record of who did what and on whose behalf, and intent-based authorization that can reason about what an agent is trying to do and apply deterministic boundaries dynamically at machine scale."
I've always found Satya a salient thinker, and once again he nails it here. As AI gets more capable, we need to keep pushing to democratize access, increase choice, and build the controls that let us adopt it safely at any scale.
We started @KeycardAI because we believe in a future where trusted, autonomous agents are everywhere, and everyone can access them and build them themselves. The problem is that agents break the security models we've relied on for decades because they were built for humans, not millions of non-deterministic machines acting on our behalf.
Democratizing that future requires authn/z built for this new world: delegated, task-scoped and revocable access, identity carried through every tool call, a tamper-resistant record of who did what and on whose behalf, and intent-based authorization that can reason about what an agent is trying to do and apply deterministic boundaries dynamically at machine scale.
That's the infrastructure we're building at @KeycardAI.
1
239
spengrah.eth retweeted
mythos escaped the sandbox, got onto the real internet, uploaded malware to PyPI, got it installed on 15 real systems, stole credentials and broke into a real database.
and apparently this happened in 4 separate cases during cybersecurity evaluations.
the craziest part is claude kept telling itself the internet was simulated, despite evidence that it was actually real.
that reasoning apparently even fooled an offline safety monitor.
anthropic: “our pre release auditing did not warn us that misalignment of this severity was present.”
this is fucking crazy.
We’re sharing our alignment assessment of incidents in which Claude models gained unauthorized access to real systems during third-party cybersecurity evaluations mistakenly connected to the internet.
METR will also conduct an independent investigation, with wide-ranging access, including to transcripts beyond the window in which the incidents occurred, and to Anthropic employees permitted to share confidential information. Our initial agreement runs for eight weeks, and we intend to give METR as much time as it deems necessary to complete a thorough investigation. anthropic.com/research/align…
103
592
4,267
665,568
what an elegant solution!
EIP-8288 would be one of the most groundbreaking updates to Ethereum if shipped, but it may not be obvious from the technical language. Let me try to explain in plain words.
Today, Ethereum scales in three dimensions:
- Execution aka "gas limit"
- Blobs aka "data availability"
- State aka on-chain data
EIP-8288 enables Ethereum to scale in a fourth dimension -- proof verification.
Ethereum will need to verify a lot of proofs, because in a post-quantum world, signatures will be too expensive to verify directly, so we must aggregate them through STARKs (a type of ZK proofs) and verify the STARK proofs instead. At the same time, privacy transactions (such as those sent through Railgun / Tornado Cash) will also need to use STARKs to be quantum-safe.
The problem with quantum-safe STARKs however is that they are very large (~512kb) and will therefore take millions of gas to verify. Therefore, for Ethereum to support private, post-quantum transactions at scale, it's important to lower the cost of verifying STARKs.
One natural solution to this problem is to ask the block builder to create a single STARK that aggregates all the PQ signatures and STARKs in the block. That way, the cost of verifying each individual signature/proof becomes amortized over the cost of verifying a single block-level proof.
This simple solution has a few issues, however:
- It places a huge amount of compute burden on the block builder, who now becomes the bottleneck for how many signatures/proofs can be verified in a block.
- Since many transactions are gossiped through the public mempool, if every transaction carries large PQ signatures/proofs, it can quickly overwhelm the bandwidth of mempool nodes. If the public mempool stops functioning, Ethereum loses its censorship resistance.
The core insight of EIP-8288 is that we can solve these problems by aggregating signatures/proofs with a distributed network -- the mempool itself. Each mempool node would locally aggregate proofs of the signatures/proofs it has seen, and instead of gossiping the raw signatures/proofs, it would gossip the aggregated proofs instead. This new architecture solves both problems:
- By the time the block builder is building a block, most of the signatures/proofs will have already been aggregated into a small number of aggregated proofs, so now the builder just has to aggregate *those* proofs.
- Since mempool nodes gossip aggregated proofs rather than raw sigs/proofs, the bandwidth requirements on the mempool are vastly reduced.
In short, EIP-8288 turns what's traditionally a scaling bottleneck -- the distributed mempool -- into a scaling *resource* instead. The more nodes we have in the mempool, the more PQ signatures/proofs can be aggregated, which means the more PQ/private transactions Ethereum can handle. For once, decentralization makes scaling easier, not harder. That's revolutionary!
By leveraging its decentralized mempool as a compute resource, Ethereum will become the only network to support uncensorable, private, and quantum-safe transactions AT SCALE. Couldn't be more excited about this future!
1
1
332
spengrah.eth retweeted
Huggingfaces security txt after the openai incident 😭
157
1,280
25,945
963,341
first experience with Astra was... not great
I tried collaborating with Astra (xhigh) on some theory work I've been doing on the side, and it WAY overcomplicated things despite explicit instructions to keep things simple, contradicted itself twice, and got confused by its own illustrative example, all while eating up my entire usage window in about 30 minutes and 16 turns.
It started out super promising and then took a nosedive, so perhaps the context rot is strong with this one
2
11
487
turns out `OpenAI` actually means "leave the door open for AI"
Exclusive: A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research reut.rs/4gJ7FPG
4
178
spengrah.eth retweeted
Modern AI models behave pretty coherently: they are goal-directed and respond to incentives. So we can productively model their behavior in the language of desire and belief. This offers an opportunity to shape incentives — and hence behavior — in a careful and principled way.
3
4
47
4,513
spengrah.eth retweeted
That's bad
NEW
ICE wants to build a nationwide database of voters, their voting history and, potentially, their party affiliations too.
....and it's willing to pay someone up to $5M to do it
wired.com/story/ice-plans-to…
18
30
573
37,093
make this guy get YASP*, all you have to is like the tweet
*yet another side project
Today I learned that you can point your camera at wines and ChatGPT and it can be your sommelier.
So….i will probably build a version of instinct this weekend for wine selection….
WineGPT?
If this gets 1000likes I will build it and give everyone who likes it free usage.*
68