AI coding tools help write more lines of code, but not ship more software: study

Do AI coding tools actually make developers more productive? It depends on how you define productivity.

After examining the data and AI usage telemetry of over 100,000 GitHub developers, researchers from MIT and UPenn’s Wharton School have determined that AI coding tools might help developers produce more lines of code, but that’s not leading to more finished software shipped.

The researchers found that synchronous agents (agents that write and edit code with the developer in real time) produced a 741% increase in lines of code and led to a 65% increase in pull requests. However, software releases only rose 20%.

Similar disparities were observed when examining developers who relied on autocomplete tools, which suggest code as the developer types.

The researchers concluded that the productivity of AI tools is weakened at later stages of software development due to “human bottlenecks.”

What’s the holdup? IT Brew caught up with Greg Jennings, VP of engineering and AI at AI-native development company Anaconda, to understand what human bottlenecks stem from leveraging AI tools.

While coding tools can accelerate code generation, Jennings said, they often don’t contribute to testing, validations, reviews, and other stages of the software development life cycle that still require human oversight and ultimately hold up production.

“What AI coding has done is it compressed the ability to write code, so now we can write code much faster, but just because you can write code much faster doesn’t mean that you can actually do a lot of the other steps faster,” Jennings said.

Sounds familiar? This isn’t the first time researchers have undermined the productivity gains of AI tools. A recent GitLab report, which surveyed 1,528 DevSecOps pros, also found that 85% believe AI has “shifted the bottleneck from writing code to reviewing and validating it.”

“Compliance, security scanning, deployment, and incident response are where AI velocity stops and the rest of the life cycle waits,” the report wrote.

IT Brew also previously reported on a research study from AI research nonprofit METR, which found some developers took 19% more time to complete assigned tasks when using AI coding tools.

What does this all mean? Jennings said the study’s findings should be an eye-opener for tech leaders, who should readjust their expectations of how AI coding tools contribute to their bottom line moving forward.

“Delivered code is not delivered value, it’s something that’s going to help us deliver more value, but now we have to pay more attention to the other steps in the process,” he said.

Manav Khurana, GitLab’s chief product and marketing officer, added it’s important for organizations using AI coding tools to invest in agentic infrastructure to support other steps of the software development life cycle: “If they don’t have the right agentic infrastructure to make sure that code created can become shipped software with the right level of automation, they also risk that the new code creates reliability problems, or security problems, or cost overruns, because speed sometimes also brings with it chaos.”