Ethereum infrastructure provider Infura is currently facing a service outage, causing a delay in price feeds of ether (ETH) and ERC-20 tokens of some services.
"We are currently experiencing a service outage for our Ethereum Mainnet API. Our on-call team is investigating and working to restore service functionality," the Infura status page reads. The issue started today at 08:12 UTC.
The outage has also caused crypto exchanges, including Binance and Bithumb, to disable ETH and ERC-20 tokens withdrawals. The popular Ethereum wallet MetaMask is also having difficulties since Infura is its default service provider. (Both MetaMask and Infura are owned by Ethereum development studio ConsenSys).
Binance CEO Changpeng "CZ" Zhao said the exchange is investigating a possible Ethereum chain split at block 11234873. "Etherscan and Blockchair are showing two different chains and data after this block. We're resolving now but have temporarily closed withdrawals," said CZ.
Nikita Zhavoronkov, lead developer at Blockchair, explained the issue in a Twitter thread. He said Ethereum developers introduced a change in the code that led to a chain-split starting from block 11234873. Those service providers who didn't upgrade their nodes, such as Infura and Blockchair, got stuck on a minority chain.
"Technically, that was an unannounced hard fork," said Zhavoronkov. "Something similar happened to #Bitcoin 7 years ago when there was a database upgrade...Fix: upgrade geth and run debug.setHead(11234872)." Geth is a tool that is used for running a full Ethereum node.
Péter Szilágyi, team lead at The Ethereum Foundation, acknowledged the issue was technically an "unannounced hard fork," from a "bad chain to the good one."
"That said, silently fixing a bug dormant for 2+ years has a much lower chance of causing a disruption than raising awareness to it. We strive to minimize potential damage," said Szilágyi.
Ethereum coder M H Swende said the Infura issue is a reminder to keep nodes up to date.
After the publication of this story, at 09:03 UTC, Infura said it has brought additional capacity online for the affected components and is continuing to assess the root cause of the outage.
The affected components include Ethereum mainnet archive data, mainnet filters, mainnet logs, mainnet WebSocket JSON-RPC API, and mainnet HTTPS JSON-RPC API. All these segments are experiencing "major" outages.
Later, at 09:47 UTC, Infura said it has identified the root cause and is preparing a fix to restore service functionality.
Some competing Ethereum infrastructure providers, including BlockCypher, QuikNode, and Alchemy, were working fine at the time of writing.
Binance resumed withdrawals at around 10 am UTC.
Paolo Ardoino, CTO of Bitfinex, told The Block that crypto exchanges should run their own Ethereum nodes. "While Infura is offering an important service to the crypto industry, exchanges should not fall into the trap of depending upon centralized parties while being in a decentralized industry like crypto," said Ardoino. (Bitfinex runs its own Ethereum nodes).
At 11:23 UTC, Infura said the root cause was traced to several components within its infrastructure, which were locked to an older stable version of the Go-Ethereum client that encountered "a critical consensus bug at block 11234873."
"This affected several Geth versions including 1.9.9 and 1.9.13. Components running 1.9.19 and later were unaffected. A full post-mortem will be completed and shared after the incident is resolved," said Infura.
© 2023 The Block. All Rights Reserved. This article is provided for informational purposes only. It is not offered or intended to be used as legal, tax, investment, financial, or other advice.