Ethereum Foundation says AI agents find real bugs, but most are false positives

The Ethereum Foundation said AI agents are good at finding vulnerabilities in the network infrastructure, but most are false positives, according to a blog posted Thursday.

"Agents finding bugs wasn't the surprise. The surprise was how little of the work went into finding them, and how much went into telling the real bugs from the ones that just looked real," the EF said.

The EF's Protocol Security team has been using coordinated AI agents to test critical network infrastructure, including systems software, cryptographic code and smart contracts. The agents have found real bugs, including "a remotely-triggerable panic in libp2p's gossipsub, a core part of the peer-to-peer layer Ethereum consensus clients run on," which has been fixed and publicly disclosed.

While the foundation said AI agents can quickly pinpoint potential vulnerabilities, the technology has simultaneously created a larger workload for human researchers, who must evaluate a growing number of potential bugs, or "candidates."

"Most candidates are wrong, duplicate, or out of scope. That's not a problem with the method; that's how it works," the EF wrote. "The goal is to reject the wrong ones fast and back the real ones with proof that's hard to argue with."

A potential vulnerability isn't considered a real finding until researchers can independently reproduce the failure against the actual code. The foundation also noted that AI agents can struggle to identify bugs that emerge across a sequence.

In other words, some of Ethereum's top security researchers consider AI to be a strong search tool, but not an oracle, so to speak.

"The time that used to go into coming up with and chasing down hypotheses now goes into judging them at scale, including building the oracle, running the triage, keeping the list of known issues, and handling disclosure," they said. "The bottleneck didn't go away. It moved from finding bugs to trusting the results, which is a better place for it, because that's where human judgment actually matters."

Thursday's blog comes on the heels of the EF's recent reorganization, which resulted in a new operational structure and the foundation shedding 20% of its total workforce.