Ethereum's consensus bug chain split revives the specter of network centralization

Wednesday’s Ethereum chain split exposed a centralized weak spot — one that developers and ecosystem stakeholders had long predicted might one day become a problem.

A post-mortem published by the Go-Ethereum (Geth) wallet development team detailed the under-the-radar effort to address a long-dormant consensus bug that, if discovered and exploited, could have led to network disruptions. That fix was included in a subsequent software release. 

The split that took place Wednesday didn’t happen because of the fix, but rather because some network participants hadn’t yet updated their Geth software to the version that included the fix. A startup created a transaction that essentially triggered the bug’s effect and nudged the nodes running the old software onto their own, separate transaction history. 

Only once the nodes were updated was the split resolved. Then, as is often the case in the cryptocurrency ecosystem, the post-mortems, the analyses, and, in some instances, the finger-pointing began. 

Long-held centralization fears

The situation was especially disruptive because one of those affected participants was Infura, a popular infrastructure provider that allows Ethereum developers to host applications without running their own full nodes. 

Infura counts a number major Ethereum services among its listed clientele, including MetaMask, Compound, Uniswap and Maker, among others. While not all of those listed clients reported problems — Compound, for example, told The Block that they weren’t affected — the chain split highlighted how a troubling situation could have snowballed into a terrifying one.

In fact, concerns about what might happen if Infura ever encountered problems date back years. Late 2018 saw discussions about decentralized alternatives to Infura, with developer Afri Schoedon declaring at the time: “If we don’t stop relying on Infura, the vision of Ethereum failed.”

The issue at the time was twofold: not only was Infura reliant on Amazon Web Services, a wide array of Ethereum-centric apps relied on Infura. The fear, as expressed by Infura co-founder Michael Wuehler: “If every single dapp in the world is pointed to Infura, and we decided to turn that off, then we could, and the dapps would stop working. That’s the concern and that’s a valid concern.” 

Two years later, Infura’s outage didn’t take down the entire network. But it did spur exchanges like Binance to temporarily halt ETH and ERC-20 withdrawals and cause disruptions for MetaMask, a popular dapp interface. 

A 'challenging and nuanced' problem

In the aftermath of the network split, some observers invoked the situation that Ethereum faced in the summer of 2016: the code exploit of TheDAO, which ultimately resulted in a hard fork and a winding-back of the transaction history. The most recent split, though very different in nature, has raised questions of centralization and whether elements of the Ethereum ecosystem had become “too big to fail.”  

Others have questioned the decision by Geth developers not to more forcefully push node operators to update their software. But in their post-mortem, the Geth team pushed back, contending that “a release that contains important consensus or DoS fixes always runs the risk of someone trying to beat updaters to the punch line and taking the network down.”

“In this particular instance, the consensus bug was dormant in the code for over 1 year,” the team continued. “The probability after all that time for someone to accidentally trigger it is tiny. Opposed to that, the probability of someone maliciously triggering it if highlighted as a security issue is not insignificant. The Geth team made the conscious decision not to mention it, hoping that people eventually upgrade to versions that contain the fix and the issue is gradually ejected from the network.”

The post-mortem concluded:

“This particular silent consensus fix took an unexpected turn with yesterday's network split, but retrospectively we still believe it was the right call. We understand that we cannot expect operators to always immediately update to the latest releases and appreciate the understanding that our vulnerability management and release structure has nuances unique to a blockchain ecosystem.”

In an interview, Nikil Viswanathan, co-founder and CEO of developer platform Alchemy, said the Geth team faced “I “a really challenging and nuanced problem.” 

“They have a really fair point. If they told people about this vulnerability, then they risk people exploiting that vulnerability,” he told The Block. 

The future

The situation has revived the age-old advice about connecting to a blockchain network: the best bet is to run your own full nodes. But there are some participants who, for whatever reason, may lack the resources or the technical know-how to do that. This is, in essence, why infrastructure providers like Infura, Alchemy and others exist in the first place. Infura and Alchemy are the largest such node infrastructure providers — one estimate contained in a report from The Block Research published this spring said that as much as 70% of Ethereum dapps are supported by Infura.

So what to do? In the case of this week’s chain split, the end result may be a more resilient — and, among service providers, cooperative — ecosystem.

Viswanathan praised the responses to the situation by the Infura and Geth teams. He told The Block that “the community, in general, would benefit a lot from more collaboration” in order to help avoid similar events in the future. 

“I think we’re going to see more communication among the players who are building and running this infrastructure,” he said. “I think we’re going to see a lot more coordination there.” 

Viswanathan also predicted that services in the space would start taking infrastructure robustness — that is, running full nodes and keeping up with software updates — far more seriously. 

“I think the companies going through this are just going to double down on being very, very vigilant on this and realize that this is something that’s a number-one priority.” 

Some of the impacted services affected indicated that they’re taking the live-fire lessons to heart and accounting for them in the development cycle.

Infura said in its post-mortem that it will “track closer to the latest client releases while still balancing the responsibility for maintaining stability and remaining backwards compatible.” Further, Infura said it will work to add more software clients to its infrastructure stuck and “review our incident response procedures and determine if there are any improvements we can implement to shorten our time to recovery.”

“This was an all-hands on-deck event that our team hasn’t seen since the network-wide attacks on Ethereum in 2016. I again want to thank our users and the community for your understanding and patience while we resolved this incident,” wrote Eleazar Galano, Infura’s co-founder and chief infrastructure engineer. 

Jacob Cantele, MetaMask’s head of product, told The Block in a message that “we learned a number of things and will be well positioned to make a number of innovations that will increase ecosystem resilience. There's nothing specific to announce yet though.”