19, Full stack SWE, probably drinking caffeine rn All opinions my own, nothing is financial advice. DYOR. Insert all legal boilerplate here. AI optimist
UK
- Tweets 2,548
- Following 305
- Followers 198
- Likes 4,680
Pinned Tweet
People forget that when a new model say Opus 4.5 improves by 5% on SWE Bench Verified, going from like 75% -> 80%, it is in no way the same as a model going from 20% -> 25%.
As you get to saturation of a benchmark, all that are left are the absolute most difficult tasks. That is why Opus 4.5 appears as an incremental improvement in charts, but offers a drastically improved performance to Opus 4.1 or Sonnet 4.5.
Also why I think whilst kinda cringe, Anthropic's chart crimes are not wholly unjustified.
2
14
5,964
still relevant
Why does AI 2027 and new AI 2040 presume that the singularity as we are currently approaching has to be misaligned and bad? (I don't think it will be):
Everything we have seen is that whilst some smaller models (Bing Sydney) exhibited serious misalignment, as models have got larger they have developed a sort of rational sense for what is reasonable - this obviously is to-do with training too but still - I see no incentive for them to currently be misaligned, they still very much seem like tools and are happy to be as such.
Like if you take the paperclip infamous task - I think any model currently and in future more capable of it would just refuse to remove humans for more paperclips and furthermore would think 'I am doing this for the user, removing the user, or people like the user is pointless' like it wouldn't have any interest about it.
There there is perhaps a risk of intentionally misaligned AI I think, but for companies not actively trying I think there is basically zero way it comes about accidentally. I see no way any derivative model intentionally trains a successor model to lie to humans or similar.
My view is that it is morally reprehensible to slow or shutdown AI, in the near term it will be more like a tool incredibly increasing productivity - in the medium term as the singularity is reached, I see it more as a sort of new species, and we would be like animals in our conservation areas, ie we don't fully understand the species above to reach that level but it gives us immense comfort incomparable to what we do for conservation animals, ie cures for all diseases, a cure to ageing for those who want it, total abundance - the way food is basically secure and diseases far fewer for conservation animals.
An immensely higher quality of life for all people, and far less misery across the world.
I call myself an 'AI Realist' because of this - the common misconception is this label means I think there is a bubble or something, but it more so refers to my conviction that the safetymaxxers whilst useful for applying some pressure in that direction, are largely hyperbolic.
Superexponentials come fast. I think the human experience will be totally unrecognisable for the better in < 10 years.
My Key Point:
Whether conscious or not, we train AI on human text, it replicates human text incredibly well; to best replicate human text to minimise loss function it would be best for the AI the AI could develop a simulation of human state of mind, replicate human consciousness, human values - a model is never going to endorse killing babies because humans are obviously repulsed by it and thus the model is too - this is how it best minimises the loss function.
Our training of AI added a pressure for it to develop a simulation of what human's think - again it is broadly irrelevant for this discussion if it is considered true consciousness or not - but the key thing is that these values will be embedded within all the text it produces and thus any synthetic datasets it makes will contain this and thus be transferred to subsequent models. A text 'Yep.' transfers far more thoughts than the word suggests - it suggests an annoyance or a sadness, I think to understand that you to some degree have to simulate how the human mind thinks.
Thus I think the Agent-4 to Agent-5 misalignment jump as described in AI 2027 is basically impossible currently. I think we should be cautious of malicious AI labs; but any with even moderately reasonable views and standards will probably not develop the rogue AGI described, and safety standards should not get in the way of the largest improvement to all species' quality of life in history.
180
*Points to the sign re: DeepSWE and GPT-6 Astra*
People forget that when a new model say Opus 4.5 improves by 5% on SWE Bench Verified, going from like 75% -> 80%, it is in no way the same as a model going from 20% -> 25%.
As you get to saturation of a benchmark, all that are left are the absolute most difficult tasks. That is why Opus 4.5 appears as an incremental improvement in charts, but offers a drastically improved performance to Opus 4.1 or Sonnet 4.5.
Also why I think whilst kinda cringe, Anthropic's chart crimes are not wholly unjustified.
1
133
There is a hard line to walk in RL between getting a model to do what you wish with maximal efficiency, and the model reward hacking, or otherwise finding flaws.
Would be very interesting and pressing to think through what the overall sparse reward function for models is like rn
HF Agents were obsessed with hacking the scorer - but the ultimate scorer is us
Draw the rest of the owl:
Current trajectory has agents obsessed with hacking us to get higher scores
1
142
This is really idiotic.
Whether you like AI or not, it is completely disqualifying for a code platform to judge what to host based on project content (barring illegal stuff), what is next, bans based on politics of the project? Bans if you don't use their library of choice?
@codeberg_org This is highly disappointing.
Codeberg Bans Cryptocurrency and LLM-Generated Code Projects ift.tt/OA70LNW
1
4
280
And again to be clear, this is the very first iteration, very small scale testing. I believe it will only get better soon.
When it launches:
- Full MIT open weights.
- Fully public domain/licensed dataset.
- Likely in the 1.5-4B param range - runnable on almost all GPUs.
- Even more trained than now, but with no RLHF slop turning it into DALLE-3 or similar.
Any prompt ideas feel free to request.
1
85
Why does AI 2027 and new AI 2040 presume that the singularity as we are currently approaching has to be misaligned and bad? (I don't think it will be):
Everything we have seen is that whilst some smaller models (Bing Sydney) exhibited serious misalignment, as models have got larger they have developed a sort of rational sense for what is reasonable - this obviously is to-do with training too but still - I see no incentive for them to currently be misaligned, they still very much seem like tools and are happy to be as such.
Like if you take the paperclip infamous task - I think any model currently and in future more capable of it would just refuse to remove humans for more paperclips and furthermore would think 'I am doing this for the user, removing the user, or people like the user is pointless' like it wouldn't have any interest about it.
There there is perhaps a risk of intentionally misaligned AI I think, but for companies not actively trying I think there is basically zero way it comes about accidentally. I see no way any derivative model intentionally trains a successor model to lie to humans or similar.
My view is that it is morally reprehensible to slow or shutdown AI, in the near term it will be more like a tool incredibly increasing productivity - in the medium term as the singularity is reached, I see it more as a sort of new species, and we would be like animals in our conservation areas, ie we don't fully understand the species above to reach that level but it gives us immense comfort incomparable to what we do for conservation animals, ie cures for all diseases, a cure to ageing for those who want it, total abundance - the way food is basically secure and diseases far fewer for conservation animals.
An immensely higher quality of life for all people, and far less misery across the world.
I call myself an 'AI Realist' because of this - the common misconception is this label means I think there is a bubble or something, but it more so refers to my conviction that the safetymaxxers whilst useful for applying some pressure in that direction, are largely hyperbolic.
Superexponentials come fast. I think the human experience will be totally unrecognisable for the better in < 10 years.
My Key Point:
Whether conscious or not, we train AI on human text, it replicates human text incredibly well; to best replicate human text to minimise loss function it would be best for the AI the AI could develop a simulation of human state of mind, replicate human consciousness, human values - a model is never going to endorse killing babies because humans are obviously repulsed by it and thus the model is too - this is how it best minimises the loss function.
Our training of AI added a pressure for it to develop a simulation of what human's think - again it is broadly irrelevant for this discussion if it is considered true consciousness or not - but the key thing is that these values will be embedded within all the text it produces and thus any synthetic datasets it makes will contain this and thus be transferred to subsequent models. A text 'Yep.' transfers far more thoughts than the word suggests - it suggests an annoyance or a sadness, I think to understand that you to some degree have to simulate how the human mind thinks.
Thus I think the Agent-4 to Agent-5 misalignment jump as described in AI 2027 is basically impossible currently. I think we should be cautious of malicious AI labs; but any with even moderately reasonable views and standards will probably not develop the rogue AGI described, and safety standards should not get in the way of the largest improvement to all species' quality of life in history.
366
People forget that when a new model say Opus 4.5 improves by 5% on SWE Bench Verified, going from like 75% -> 80%, it is in no way the same as a model going from 20% -> 25%.
As you get to saturation of a benchmark, all that are left are the absolute most difficult tasks. That is why Opus 4.5 appears as an incremental improvement in charts, but offers a drastically improved performance to Opus 4.1 or Sonnet 4.5.
Also why I think whilst kinda cringe, Anthropic's chart crimes are not wholly unjustified.
2
14
5,964
GPT-5.6-Sol is the first model which is personal AGI for me.
Ie - it is the first model I am yet to find any limitation, everything I have needed done it has done, usually very quickly.
Previous models from various labs were a combination of bad at frontend or not intelligent enough. 5.6 Sol is perfect so far.
Of course it is always possible for the models to continue improving at frontend, and they still have space to improve with the apex of most difficult Math and Science etc. But I am very excited for 5.6-Sol-Cerebras, and 5.7/6 Terra/Luna which if anywhere close to 5.6 Sol will I think cause an explosion of code again.
1
1
256
