I belong to Jesus. Core developer #MultiversX
- Tweets 15,492
- Following 726
- Followers 22,792
- Likes 18,817
Not bad after the restart.
blocks, consensus, processing working again like magic.
And we never stop building.
⚡️ @MultiversX just hit a new TPS record
The network reached 21,798 TPS on September 24, setting a new peak for its throughput 👀
21K+ TPS is a serious benchmark for any blockchain
📊 chainspect.app/chain/multive…
9
38
238
4,056
today's news.
all week long it was about MvX recovery.
it was a hard week, that is for sure.
work did not stop, new things are merged into main / master. cleanups and new code too.
agents are fired up for more verifications, writing more invariants at critical paths, more verifications, harden more.
AI brought us an interesting year. yes, it is pretty obvious that a lot of hackers are trying to constantly steal, not only on MvX, but absolutely everywhere. from Apple, to various chains, to hot wallets, to servers, to governments.
and right now the only way to do better is to work faster and plug the wholes, make the code safer and faster with the same or better AI agents and models.
this is a constant race. let's win it.
we have to win it, there is no other choice.
9
42
222
3,283
this is a pretty impressive video.
humanity is capable of amazing things.
humanity is not doing good, only when it is forcefully stopped or paused.
we were created in the image of God, who actually created the whole universe. and yes, he gave the authority and actually the command to discover the world. not only map it, but to discover more and more of the laws and to create.
and why are we still not extinct? and why do we have so many miracles which stopped the super bad things at the exact right time? because God loves us.
real freedom is accepting Jesus and after creating what is good, doing and trying what is good.
AI is not going to kill us, fire did not kill us, nuclear did not kill us. but in the 20th century what killed most of the people were socialist/communist regimes, which stole the freedom from people.
it is always time to build.
4
3
49
2,682
the chain is back.
it was a force major, an exploited vm endpoint with the combination of a built in function. vm though built in function is atomic and it was not. technically both of them were written by me, audited multiple times, formally verified too.
the exploit was in a code which entered in the mainnet at barnard release. we run import db since that release and no other exploits were recorded. the exploiting wasm contract was purposefully built for this, writing directly wasm, not using the mvx SDKs.
what we did actually to restore the chain?
we made a hardfork, technically restarting the chain with the new code from a moment before the hack. in this way, no tokens were minted, the hack simply did not happen from the perspective of the chain's history.
all the processes are slowly being restarted, we are monitoring all the dapps, safeprice and until now, everything seems fine.
we made this hardfork code without any blacklist / whitelist / centralisation aspect. all the coordination with validators was in the telegram channel, and everyone supported it. exchanges too, they support this hardfork.
these were a couple of super hard days.
thanks for everything.
MultiversX is back online.
Thank you for your patience and belief through these hard days.
To our validators and partners: thank you for the work that brought us here.
We’re proud to be here with you.
We continue to build different. Together.
36
65
309
8,582
THE CHAIN IS BACK.
all shards and metachain are notarising and working.
monitoring all from the war room.
things are catching up. blocks are produced.
WE ARE SO BACK!
p.s. a big thing to get from this, it is time to write a new client, directly in rust, via specs and formal verification first. it will be one of my side projects, agents building it in the night, 24/7, it will take some time.
After carefully reviewing yesterday’s logs, our engineering team confirmed that the recovery hard fork completed successfully. It does not need to be repeated.
Validators are now upgrading and synchronizing for the coordinated restart. Please continue to wait for the separate all-clear before submitting transactions.
Thank you to our validators, partners and everyone standing with us through these difficult days.
We keep building, together.
51
96
426
12,999
thanks for getting in deeper in deeper.
it is definitely wrong that it was left there.
it was definitely not pressed. nobody has the key. it is stupid to leave it there.
was I wrong with the first assessment? yes. I thought that part was fully invalidated from the code at one point. and this back and forth proved that we have one more bug on the node binary.
i did know that a lot of the validators, configs, our own machines had that API off, as this said.
i wanted to demonstrate that we did not use, nobody used such a thing. we are not malicious, centralised.
in the other hand, a few people framed it as centralised off switch button, and some people, who actually got to meet the team, frames everything as mvx people are liars and malicious. that is what people feel when that message is read. whether it is meant to do so or not, i do not know.
the problem is that there are some who come only when something is bad, and one side is presented, the rocks is thrown and that is it.
it would be a totally different framing: legacy code is there, i asked validators, everyone stopped their own nodes.
but yeah, this part is not written.
back to monitoring the restart.
I got tired of watching people announce opposite views from the same repo.
Reading the code is useful. Running it is better. So I turn up a private local chain and tested the off switch myself.
Two doors.
The local command is dead on mainnet. 404.
The gossip path still works, one signed message stopped a node. Then it stopped three.
That is not what halted mainnet on Sep 19. I was in the validator chat. Operators stopped their own machines by hand.
The switch existing and the switch being pressed are different claims. @Justin_Bons was right that it exists. @SasuRobert was right about the channel, and wrong that there is no off switch. @vinibarbosabr already put the files on the table.
I pressed the button to verify it all...
@MultiversX @CodeMultiversX
11
16
112
4,624
It seems like we are MOVING.
New release and block production to resume in a few hours.
today we will monitor the logs and graphana and everything.
war room is still on.
so yesterday's attempt went well.
we were too tired and graphana, our internal monitoring tool showed us wrong data and we thought the chain is not moving, shard 1 is not moving.
and actually it was. with less validators and most of the times it could gather only 200-240 votes for the block, and on other cases, when it passed it was just 270. (you need 267 votes per block).
tiredness, stress, everything accumulated made us to message validators to stop the nodes and we did not had time to check the logs, we were thinking that the chain can get in other impossible scenarios.
however, the chain, consensus, the code was working amazingly, even in this super bad scenario.
today is another day.
validators are organising in telegram for the restart.
10
44
212
4,851
posted my reply to an audit in which i am called liar countless of times. and yes, evidently if people read what vini says, it writes that he is not claiming that it was used, but the whole framing is the one in which he wants to discredit me and multiversx.
in the config is a key from 2020 , nobody has that.
it is a code from 2020, nobody tested it.
we changed P2P libraries since that, we have changed how signatures are verified, we have changed how messages are propagated, we did a bunch of things.
is it that hard to understand that there was no ill intention, no centralised switch off, nothing of that sort.
how hard it is to understand that we develop a lot and forgot something there which was not working.
and yet, i am being portrayed as a liar.
why am I even bothering.
it is saddening when people who call themselves righteous make these attacks because they have vendetta against mvx ...
like literally. i am so tired about these.
and in the meantime I have posted various AUDITs on why that CODE is not working and it was never working.
like I go to you friendly, explain things, and you are still framing it in such a way that multiversX and myself are liars. why am I even bothering ...
all the validators have spoken. there was no off switch. it was a forgotten code which is not working from P2P level.
validators are not accepting messages like that, we changed a lot on P2P level.
anyway. it just makes my head hurt.
like really, coming friendly and making such a report.
sorry. but this was another, how do you call it, low ball.
I can disprove those things.
come on. launch a testnet fully and verify. ASK the logs for all the validators and verify.
but it is stupid. like I am doing good, actually responding to your request to delete legacy code, and I am portrayed as a liar.
the code tells the story. it simply does not work.
and i stand by my claims, it does not work, never worked, it was never intended to be used, nobody has the private key for that setup public key.
P.S. it is clear some has just one agenda, destroy mvx.
31
26
136
5,285
Viewed trigger.go:320-370
Searched for "AddCloser"
Viewed exportHandlerFactory.go:1-100
Viewed exportHandlerFactory.go:200-340
Viewed exportHandlerFactory.go:340-420
Viewed exportHandlerFactory.go:421-500
Viewed exportHandlerFactory.go:500-600
Searched for "ChanStopNodeProcess"
Searched for "chanStopNodeProcess"
Viewed nodeRunner.go:1525-1560
Searched for "CreateManagedCoreComponents"
Viewed nodeRunner.go:270-320
Viewed nodeRunner.go:150-270
Viewed nodeRunner.go:450-520
Viewed nodeRunner.go:521-600
Searched for "func waitForSignal"
Viewed nodeRunner.go:1028-1120
Viewed nodeRunner.go:1120-1200
Searched for "func closeAllComponents"
Viewed nodeRunner.go:1630-1720
Viewed node.go:1040-1100
Searched for "func (n *Node) Close"
Viewed node.go:1428-1500
Searched for "SetStartImport"
Viewed importStartHandler.go:75-115
Viewed importStartHandler.go:1-75
A thorough inspection of the execution flow in [`update/trigger/trigger.go`]
1. **It does NOT close the node** (it leaves the node half-alive in a zombie state).
2. **It CANNOT save everything** (it omits critical modern state and exports empty root tries).
3. **It causes severe structural and database damage** if triggered.
Here is the exact step-by-step reality of what happens in the code:
---
### 1. Does It Actually Close / Kill the Node? **No.**
When the trigger fires, it calls [`doTrigger()`]update/trigger/trigger.go#L321):
```go
func (t *trigger) doTrigger() {
t.callClose()
t.exportAll()
}
```
#### What `callClose()` actually does:
`callClose()` only closes the slice `t.closers`. Across the entire codebase, only **5 consensus objects** are ever registered as closers:
1. `cc.worker` (the block production/validation worker)
2. `cc.chronology` (consensus round clock)
3. `cc.bootstrapper`
4. `coreComponents.SyncTimer()`
5. `nodesShufflerOut`
#### What `callClose()` does **NOT** close:
* **The Databases**: LevelDB / Trias DB, state tries, block storage, and miniblock stores remain **completely open**.
* **The P2P Network**: The libp2p messenger, open network sockets, topics, and peer connections are **NOT closed**.
* **The API / WebServer**: The HTTP server keeps running and serving endpoints.
* **The Interceptors & Mempools**: Transactions and blocks continue to be received and pushed into pools.
* **Async Execution Manager & VM**: The WASM runtime and execution managers are **NOT closed**.
#### If Export Fails (Guaranteed on Mainnet):
Inside [`exportAll()`] update/trigger/trigger.go#L334-L346):
```go
err = exportHandler.ExportAll(epoch)
if err != nil {
log.Error("error while exporting data", "error", err)
return // <--- IT SIMPLY RETURNS!
}
```
If `ExportAll` errors out:
* It **never** signals `chanStopNodeProcess`.
* It **never** calls `node.Close()`.
* **The process never terminates.**
* The node is left running indefinitely as a **zombie**: its consensus worker is dead (it will never produce or sign blocks again), but all background networking and DB handles remain open.
#### Even If Export Succeeded:
Look at [`update/trigger/trigger.go:354`]update/trigger/trigger.go#L354):
```go
wait := time.Duration(t.closeAfterInMinutes) * time.Minute
time.Sleep(wait)
t.chanStopNodeProcess <- argument
```
In `cmd/node/config/config.toml`, `CloseAfterExportInMinutes` is set to **`10000`** (which is **almost 7 days**!). The node would sleep for a week with broken consensus before ever sending the stop signal to `nodeRunner`.
---
### 2. Why It CANNOT Save Everything
A proper node shutdown ([`closeAllComponents`](mx-chain-go-private/node/nodeRunner.go#L1631)) blocks pruning, waits for round persistence, flushes dirty memory caches, and flushes account tries to disk.
The hardfork trigger does none of that:
1. **Uncommitted Memory State is Lost**:
Because `managedStateComponents.Close()` is never called prior to export, dirty trie nodes and uncommitted accounts in memory are **never flushed to disk**.
2. **Supernova Root Hashes are Nil (`BUG-701`)**:
In [`update/sync/syncAccountsDBs.go:113-118`](update/sync/syncAccountsDBs.go#L113-L121), the exporter queries:
```go
meta.GetRootHash()
meta.GetValidatorStatsRootHash()
```
On modern `MetaBlockV3`, those methods return **`nil`** (roots exist only inside `ExecutionResults`). As a result, the exporter attempts to sync tries from `nil` roots, **exporting an empty state trie** (0 user accounts, 0 validator accounts).
3. **Dropped In-Flight Transactions (`BUG-712`)**:
Because Supernova decouples proposal from execution, referenced miniblocks in `HeaderV3` are marked `notPending`. The exporter does not drain `ExecutionResults`, causing all committed-but-unexecuted transactions at the boundary to be permanently dropped.
4. **Missing Modern Protocol State**:
The exporter does not serialize or export:
* Migrated data trie status (`is-data-trie-migrated`)
* Scheduled miniblocks & delayed execution queues
* Modern smart contract storage mappers and multi-ESDT balances
---
### 3. Structural & Database Damage to the Node
Triggering the hardfork inflicts serious damage on the node’s filesystem and DB layout:
1. **Destructive Folder Reset**:
In [`update/factory/exportHandlerFactory.go#L569), the export factory executes:
```go
err := os.RemoveAll(folder)
```
If the export folder overlaps with any active data or is misconfigured, it recursively deletes that directory from the filesystem while the node is still active.
2. **Concurrent Database Contention**:
While the node's main storage engines are still active and accepting P2P data, the export factory attaches brand new syncers and storers directly against the active `StorageService` and `DataPool`. This causes severe IO thrashing, cache invalidation, and race conditions on LevelDB.
3. **Corrupted Replay Without Rollback (`UF-062`)**:
In [`update/process/processPending.go:78-91`](update/process/processPending.go#L78-L91), pending item reconstruction does not implement rollback snapshots. If a transaction execution fails during the export replay, state mutations remain committed to the export DB, while the transaction hash is omitted from the miniblock, resulting in non-deterministic state divergence.
4. **Poisoned Restart Marker (`mustimport`)**:
If export gets far enough, it writes a `mustimport` file in the working directory ([`importStartHandler.go:85`](update/trigger/importStartHandler.go#L85)). If the operator attempts to restart the node normally, the node detects this file, enters import mode, and refuses normal startup.
---
### Summary Table
| Requirement for a Safe Hardfork | What the Code Actually Does | Verdict |
| :--- | :--- | :--- |
| **Clean Node Shutdown** | Calls `Close()` only on 5 consensus objects; leaves DBs, P2P, and API running. | ❌ **Leaves node in a zombie state** |
| **Flush Dirty State to Disk** | Never calls `managedStateComponents.Close()`; in-flight state is abandoned. | ❌ **Data loss** |
| **Export Modern Ledger State** | Calls `meta.GetRootHash()` which returns `nil` on `MetaBlockV3`. | ❌ **Exports empty tries (0 accounts)** |
| **Terminate Process** | Logs error and returns; if successful, sleeps for 10,000 minutes (7 days). | ❌ **Process never exits cleanly** |
| **Database Integrity** | Spawns secondary syncers that contend with live DBs; lacks execution rollback. | ❌ **Severe DB & state corruption** |
**the mechanism cannot cleanly shut down the node, cannot capture modern state, and would leave the node's DB and process in a crippled, corrupted state.**
and some people claim we have a kill switch.
the reality, we have forgotten a stupid code in the node level which is not working, even if it would, it would create insane damages at all levels.
1
15
505
so yesterday's attempt went well.
we were too tired and graphana, our internal monitoring tool showed us wrong data and we thought the chain is not moving, shard 1 is not moving.
and actually it was. with less validators and most of the times it could gather only 200-240 votes for the block, and on other cases, when it passed it was just 270. (you need 267 votes per block).
tiredness, stress, everything accumulated made us to message validators to stop the nodes and we did not had time to check the logs, we were thinking that the chain can get in other impossible scenarios.
however, the chain, consensus, the code was working amazingly, even in this super bad scenario.
today is another day.
validators are organising in telegram for the restart.
we need some new code.
we need to review, audit, verify and test it.
the developers are on. we are working.
preparing a new release.
but it will remain pending for tonight.
We are preparing a new release for the next recovery attempt. An epoch change occurred during today’s attempt, so we need to test recovery across that epoch change before proceeding.
we thought there would be quorum on all shards before epoch change, shard 1 missed that window.
coding, testing, coding, testing.
26
40
186
11,466
we need some new code.
we need to review, audit, verify and test it.
the developers are on. we are working.
preparing a new release.
but it will remain pending for tonight.
We are preparing a new release for the next recovery attempt. An epoch change occurred during today’s attempt, so we need to test recovery across that epoch change before proceeding.
we thought there would be quorum on all shards before epoch change, shard 1 missed that window.
coding, testing, coding, testing.
Update:
The Mainnet restart is still pending.
Shard 1 did not complete the recovery transition in time, preventing its blocks from being confirmed by the metachain. Validators have been asked to stop their nodes while engineering assesses whether the restart can be retried immediately.
Please continue to wait for the all-clear: do not submit or rebroadcast transactions, or use EGLD and ESDT deposit and withdrawal routes through exchanges or bridges.
We will share the next confirmed steps as soon as they are available. Thank you for your continued patience and support.
22
29
194
13,493
shard 0, shard 2, metachain has started.
waiting for shard 1.
that is the hardest, as it has the biggest trie.
watching it live from the war room.
shard 1 not yet. monitoring. checking if there is consensus. it might be only a few more seconds or minutes.
more validators need to be synced for shard 1.
things are looking good.
and if you go and check all the code, NO NODE is trusting blindly anything. they are validating absolutely everything with their own data.
they are going back to the given round and they validate signatures, roothashes, validate the trie, resync the trie if needed, with full validation.
there is no trust, but verification of every single step.
no whitelists, no blacklists, no switch keys, nothing.
it took more time as we wanted to do it correctly, fully, all in verifications in all steps on all nodes.
this is where blockchain is different from anything.
now validators will upgrade and at the setup time they will start producing blocks again in around 2 hours.
16:35 UTC.
the clean way to organise a hardfork.
7
37
190
9,896
shard 1 makes something, but metachain is not accepting. meta went over new epoch and shard 1 did not get in time.
let's see if we can do something.
checking logs live.
5
4
52
2,099
we need to reassess and redo.
that is the reality.
shard 1 missed its time before the epoch change and the time after.
shard 1 did not get enough validators synced for the actual block production.
so the validators did not have enough time to validate and sync everything.
p.s. it proves there is no centralised buttons.
a long hard night ahead.
6
8
95
1,832
things are looking good.
and if you go and check all the code, NO NODE is trusting blindly anything. they are validating absolutely everything with their own data.
they are going back to the given round and they validate signatures, roothashes, validate the trie, resync the trie if needed, with full validation.
there is no trust, but verification of every single step.
no whitelists, no blacklists, no switch keys, nothing.
it took more time as we wanted to do it correctly, fully, all in verifications in all steps on all nodes.
this is where blockchain is different from anything.
now validators will upgrade and at the setup time they will start producing blocks again in around 2 hours.
16:35 UTC.
the clean way to organise a hardfork.
mainnet upgrade is near. war room is still on.
a lot of verifications on and on and on.
follow the progress.
12
36
163
10,341
mainnet upgrade is near. war room is still on.
a lot of verifications on and on and on.
follow the progress.
Update:
The Devnet recovery rehearsal has completed successfully. Following successful recovery runs on both Testnet and Devnet, we are proceeding with Mainnet recovery today, Wednesday, September 23.
The planned validator upgrade and synchronization window is 14:00–16:30 UTC today. This preparation period allows participating nodes to become ready before block production resumes.
We currently estimate that Mainnet will begin producing blocks at approximately 16:35 UTC, subject to successful completion of the recovery process and validator readiness. This remains an estimate, not a confirmed restart time.
Mainnet remains paused, and a separate all-clear will be issued for users. Until then, please do not submit or rebroadcast transactions, or use EGLD and ESDT deposit and withdrawal routes through exchanges or bridges.
We appreciate your patience as we work to restore Mainnet safely. Thank you to the validators, exchanges and infrastructure partners working with us on the recovery.
9
18
158
6,662
unhinged history from validators. this is the reality. it is all about zeros and ones.
in the meantime the devnet hardfork rehearsal is happening.
last step before mainnet.
and coordination is via the telegram channel with validators and separately with exchanges and integrators.
There’s a lot of talk right now about MultiversX downtime, off-switches, first halt, all of it.
Fine. Let’s go through what actually happened since launch, six years ago, from a validator’s chair.
I’ve run a node since day one. Before it was MultiversX. When the chain was still Elrond. You remember the weekends more than the marketing. The nights Telegram lights up. That’s the job.
People talk this week like the chain never broke before, or like someone pressed a button on my servers. Neither is true.
December 2021 — RIDE
Nobody asked us to stop nodes. The chain kept making blocks. Listing traffic, bots, timeouts. Unusable for hours. Congestion.
Consensus downtime: none. I didn’t reboot the server.
June 2022
multiversx.com/blog/incident…
VM bug. executeOnDestContextByCaller in a callback. Same shard. ~1.65M EGLD out of the wrap contracts, dumped on Maiar.
They paused DEX, WEGLD swap, bridge, public API send. Not the chain. The report says the blockchain was never turned off. Own API still worked. In-place upgrade, most nodes on the new binary in ~30 min. Users made whole.
Consensus downtime: none. Apps dark ~2 days.
May 14, 2025
multiversx.com/blog/sc-bug-i…
First real consensus halt.
transferValueAndExecute. Missing check. Huge wrapped USDC credit, then USDT. They called verified / red-line operators, community delegation, 25+ SPs. Enough to break 66%. Quorum died. Patch v1.8.13.0. Back in 2h30. 99.999% recovered. No users directly affected.
I wasn’t on that call. If you weren’t in that group you just watched it stall.
Consensus downtime: 2h30.
September 19, 2026
nitter.net/MultiversX/status/2101…
Same idea, bigger.
VM atomicity. Invalid state already on chain. Official line: order-of-operations on an edge case. This time they asked everyone to keep mainnet nodes down. Hard fork from a checkpoint. That’s why it’s days, not two hours.
I stopped my own nodes after the announcement. Nobody flipped a switch on my servers. I can still refuse the upgrade. PublicKeyToListenFrom in my config.toml is empty. I never cleared it.
Should be back this afternoon. Call it ~4 days. ~96h.
The map
2021 RIDE: consensus 0
2022: consensus 0, apps ~2 days
2025: 2h30
2026: ~96h, restart this afternoon
SLA since launch:
Since genesis (July 2020.) : ~98.5h down → about 99.82% uptime.
Since 1 Jan 2026: only this one → about 98.5% this year.
We’ve had incidents. We’ve had downtime. This is just the first time it stayed down more than a few hours. Not a fairy tale about a button.
Those are the facts.
4
25
110
3,610
so to not create any confusion, people can accuse whatever they want. all the AI said update folder is dead code. some sync files were moved, as they are used in epoch start process.
and everything required, so that this should not be in a discussion anymore.
if anyone wants more confirmation that it was not used, it can ask any of the validators for logs. or if he had nodes, observers or anything it is ultimate proof that it was not used.
technically the ultimate proof is that at the p2p messaging layer, any kind of message constructed with this old legacy code would be dropped by the validators.
this is not done to erase traces or whatever, as some might try to twist in that direction, it is a totally sincere response, to clean up any kind of confusions.
p.s. on other chains from configs, they can blacklist / whitelist addresses, transactions, and more crazy configs exist out there. every such config can be written by the validator as he wants. and those chains are decentralised too. it is not like node level hardcoded things which respond to a central command, not even on those other chains.
this is for total transparency, this code will not enter in todays restart, as it needs internal testing and the whole release procedure. but it is prepared in a PR and it will get in as soon as possible.
7
35
150
7,565
and the PR: github.com/multiversx/mx-cha…
9
61
3,595