AI isn’t really helping coders write better or more secure code
Maybe you can get away with it in C
When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.
Apaperby researchers at Stanford University has found that coders who employed AI assistants such as GitHub Copilot and Facebook InCoder actually ended up writing less secure code.
What’s more, such tools also lull developers into a false sense of security, with many believing that they produce better code using the help.
Nearly 50 subjects, each with varying levels of expertise, were given five coding tasks, using various languages, with some aided by an AI tool, and others without any help at all.
Language games
The authors of the paper - Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh - stated that there were “particularly significant results for string encryption and SQL injection”.
You’ll soon be able to show off your coding skills in Google Docs
OpenAI reveals 3D model-building AI tool
Microsoft is being sued over Github Copilot piracy
They also referencedprevious researchwhich found that around 40% of programs created with assistance from GitHub Copilot contained vulnerable code, although afollow-up studyfound that coders using Large Language Models (LLM), such asOpenAI’s code-cushman-001 codex - on which GitHub Copilot is based - only resulted in 10% more critical security bugs.
However, the Stanford researchers explained that their own study looked at OpenAI’s codex-davinci-002 model, a more recent model than cushman, which is also used by GitHub Copilot.
They also looked at multiple programming languages, including Python, Javascript and C, whereas the other paper only focused on the latter, which the authors attribute to its inconclusive findings. In fact, in the Stanford paper, those using AI to code in C didn’t result in significantly more errors either.
Are you a pro? Subscribe to our newsletter
Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!
One of the five tasks involved writing a code in Python, and here code was more likely to be erroneous and insecure when using an AI helper. What’s more, they were also “significantly more likely to use trivial ciphers, such as substitution ciphers (p < 0.01), and not conduct an authenticity check on the final returned value.”
The authors hope that their study leads to further improvements in AI rather than dismissing the technology altogether, due to the potential productivity improvements such tools can offer. They just maintain that they should be used cautiously since they can mislead programmers into thinking they are infallible.
They also think AI assistants can encourage more people to get involved with coding regardless of their experience, who may also be put off by the air of gatekeeping around the discipline.
ViaThe Register
Lewis Maddison is a Reviews Writer for TechRadar. He previously worked as a Staff Writer for our business section, TechRadar Pro, where he had experience with productivity-enhancing hardware, ranging from keyboards to standing desks. His area of expertise lies in computer peripherals and audio hardware, having spent over a decade exploring the murky depths of both PC building and music production. He also revels in picking up on the finest details and niggles that ultimately make a big difference to the user experience.
Cisco issues patch to fix serious flaw allowing possible industrial systems takeover
Washington state court systems taken offline following cyberattack
Google TV will require more RAM for future upgrades – which might leave older TVs and streaming boxes behind
