Medalla encounters continued headwinds after Eth2 testnet’s weekend turbulence

It’s been a bumpy few days for Medalla, the in-progress Ethereum 2.0 testnet.

As The Block reported last week, the testnet serves as a dress rehearsal of sorts for the eventual launch of the next-generation version of Ethereum. The existing mainnet launch window lies somewhere between the end of 2020 and early 2021, and the multi-client Medalla is intended to provide an as-close-to-the-real-thing as-possible perspective on how the current design works in practice. If bugs are to emerge, the idea is to have them crop up first while running the testnet — which first went live on August 4 — rather than when the real show begins.

That is exactly what happened. In this case, the problem stemmed from a bug in Prysmatic Labs' Prysm client. Essentially, every node operates by its own internal clock, and its the alignment of those times that keeps everything moving in sync. The flaw caused these clocks to desynchronize, triggering what Prysmastic Labs’ Raul Jordan described as “a series of cascading failures.” That caused the network to partition, effectively stopping Medalla in its tracks.

As a breakdown of the incident from Prysmatic Labs notes:

“Starting with the receipt of bad responses from 6 different time servers which threw off most nodes running our Prysm client at the same time, our team rushed to push a fix to the problem. This fix contained a critical flaw which removed all necessary features for our nodes to function. This problem led to network partitions, with everyone synchronizing the chain at the same time but unable to find a healthy peer, Medalla had a very eventful weekend and offered us the greatest learning experience to prevent this happening again, especially on mainnet.”

As a result, the number of participating validators — crucial for the operation of the network — plunged, as shown by the publicly available data from Beaconcha.in.

A low participation rate means that finality — essentially a guarantee that transactions can no longer be reversed — is unable to be achieved. As the chart above shows, Medalla struggled to reach a point wherein finality could be had on the network. The issue has inspired calls on social media to “help save Medalla" by encouraging people to get their validators moving again.

But recent developments on the Discord server run by Prysmatic Labs suggests that after Medalla re-achieved finality around 2:40 p.m. ET, new problems emerged among node operators trying to sync their nodes to the beacon chain. Prysmatic Labs co-founder Raul Jordan wrote on Discord at 4:23 p.m. ET that "there was no disagreement on the finalized block between nodes. The peer disconnections were due to a different bug at the time of finality that had to do with inefficiencies in processing at the database layer. There is a fix in progress in prysm and in teku as we speak, hope this helps clarify."

PegaSys engineer Ben Edgington wrote in his own breakdown of the problems on August 17 that among the lessons learned is that “[m]ass reliance on third-party time services is a serious vulnerability for the network.”

Edgington said the incident demonstrates that stakers who are participating in the testnet should play a more active role. “For some time yet, there will be a need for those staking to pay attention, read the forums, provide feedback to devs, and update their clients at short notice," he wrote. "I am strongly supportive of individuals getting involved and running their own validators, but best to be aware of the commitment ahead of time.”

As for what might happen should Medalla not recover from the situation, Edgington suggested that "even if we could keep the network running, then we might think about restarting from scratch, and a fresh deposit contract (it might be good to rehearse genesis again)."

"But that’s just a backup option at this stage," he continued. "Viva Medalla!"