Ethereum consensus layer client Prysm releases post-incident analysis report on the Fusaka mainnet failure.
TL;DR
Prysm released an analysis of the Fusaka mainnet outage, where nodes exhausted resources processing faulty proofs, causing 248 lost blocks and ~382 ETH in validator losses. The issue was resolved with a temporary parameter fix and permanent updates in versions v7.1 and v7.1.0.
Tags
ChainCatcher reports that the Prysm team, the Ethereum consensus layer client, has released a post-incident analysis report on the Fusaka mainnet outage. During the incident, almost all Prysm nodes experienced resource exhaustion while processing specific proofs, resulting in their inability to respond to validator requests in a timely manner. The outage affected epochs 411439 to 411480, resulting in the loss of 248 blocks over 42 epochs, a missing block rate of 18.5%, and a minimum network participation rate of 75%. Validators lost approximately 382 ETH in proof rewards.
The root cause of the problem was that Prysm beacon nodes received proofs sent by potentially out-of-sync nodes, which referenced the block root of the previous round. To verify these proofs, Prysm attempted to rebuild a compatible state, resulting in the reprocessing of past round blocks and costly recalculation of round transitions. The team temporarily resolved the issue by instructing users to use the `--disable-last-epoch-target` parameter; a long-term fix was included in subsequent versions v7.1 and v7.1.0.