Prysm, the consensus client for Ethereum, announced that validators lost 382 ETH (worth over 1 million dollars) due to a software bug that caused network disruption immediately after the Fusaka upgrade.
Details of this incident were revealed in a post-mortem report titled "Fusaka Mainnet Prysm incident." Resource exhaustion occurred in nearly all Prysm nodes, resulting in numerous blocks and attestations being missing.
What caused the Prysm stop?
According to Offchain Labs, which developed Prysm, the issue surfaced due to a bug that had existed before, causing delays in validator requests on December 4.
This delay resulted in the loss of many blocks and attestations across the entire network.
The project explained that "the Prysm beacon node received attestations from nodes that may not be synchronized with the network. These attestations referenced the block root from the previous epoch."
This failure resulted in the loss of 41 epochs, with 248 blocks missing out of 1344 slots. This indicates a 18.5% slot loss rate, and during the failure, the overall participation rate of the network dropped to 75%.
According to Offchain Labs, the bug that triggered this behavior was introduced to the testnet about a month ago and occurred on the mainnet after the Fusaka upgrade.
While immediate significant impacts were mitigated by temporary relief measures, Prysm implemented permanent changes to the attestation verification logic in response to this incident to prevent recurrence.
Distribution of Ethereum clients
On the other hand, this incident highlighted the risks brought about by the concentration of Ethereum clients and a single software culture.
Offchain Labs pointed out that if Prysm's share had been higher in the Ethereum validator base, the impact could have been more severe. The company stated that the diversity of Ethereum clients was crucial in preventing the expansion of failures across the entire network.
It was pointed out that "if the client occupies more than one-third of the network, there is a possibility of temporary finality loss and further block loss. If a client with bugs occupies more than two-thirds, there is a risk that an invalid chain could be finalized."
Nevertheless, this case has further heightened the necessity for client diversification.
According to data from Miga Labs, Lighthouse remains the main Ethereum consensus client, accounting for 51.39% of validators. Prysm is at 19.06%, Teku at 13.71%, and Nimbus at 9.25%.
The share of Lighthouse is approximately 15 points away from the threshold considered systemic risk by researchers.
As a result, voices are rising from developers and ecosystem stakeholders urging validators to migrate to other clients to mitigate the risk of disruption to the core functionality of the blockchain due to a single software defect.
