Can ChatGPT Really Replace Crypto Audit? Not now, say researchers – Decrypt
Everyone is experimenting with ChatGPT, even crypto exchanges.
Coinbase recently turned to artificial intelligence to experiment with how ChatGPT could conduct a token security review – a requirement for all tokens listed on the exchange.
After reviewing 20 different smart contracts, the mega-popular AI tool produced 12 times the same results as a manual review.
However, five of the eight defaults were cases where ChatGPT incorrectly labeled a high-risk asset as low-risk, a worst-case failure.
The experiment also showed that the AI sometimes produced inconsistent results, with the same signal generating different results, especially when moving from one iteration of ChatGPT to the next.
Nevertheless, the Coinbase team is optimistic that – with further prompt engineering – they can increase the accuracy of ChatGPT to the point where it can be used as a secondary quality assurance check.
“We are not surprised because such smart contracts can also be automatically audited [other traditional programming] equipment,” explained a spokesperson for BlockSec, a blockchain security infrastructure firm. decrypt, “However, this may not work for complex business logic, which are the main attack surfaces and the main loopholes smart contract audits should focus on.
Coinbase’s optimism about using the tool for additional reassurance was nonetheless echoed by other security experts in the crypto-security space.
“At this stage, it [AI] cannot replace a person, but it is an essential aid, including for tired or inattentive auditors,” notes the independent security researcher Officer decrypt via twitter. “I think Q/A [quality assurance] And fuzzing certainly won’t do without AI tools in the future.”
ChatGPT is replacing engineers
Although tentative, the blockchain security sector is embracing the potential implementation of AI tools.
But can AI replace manual security auditors in the future?
“Perhaps one day we will get to that point, but we are still a long way off. What is more likely a complementary approach. There are some things humans do better than machines and vice versa,” Sertic’s solutions Head of Architecture Connie Lamm told decrypt Via email, “Tools help us create new things, but they don’t replace us. The invention of the calculator didn’t make accountants obsolete, it made them better at their job.”
For now, though, non-AI security tools are still far more useful than entering the market to find vulnerabilities.
A spokesperson for OpenZeppelin said, “Existing security audit tools are still better than OpenAI.” decrypt, “They [Coinbase] Testing it out for listing ERC-20 tokens is a well known pattern. This makes it more suitable for automation.”
Although this may change.
Rapid progress can also be seen between ChatGPT 3.5 and ChatGPT 4, suggesting further upgrades will continue to impress.
And as they do, integrating these tools should be “encouraged”.
Lam said, “The use of ChatGPT should be encouraged during the development phase. It is a powerful tool, and to refuse to work with it and learn what it can do would be a blow.” “ChatGPT is also a very powerful tool for discovering information and building a knowledge hub. It can help users quickly understand complex topics and stay up to date on the latest security information.”
Stay on top of crypto news, get daily updates delivered to your inbox.