The AI tools industry execs say that can make you more productive may actually be holding you up.
That’s according to a recent study by AI research nonprofit METR, which found that experienced developers who used AI tools in their workflow took 19% more time to complete assigned tasks.
Prove it! The researchers conducted a randomized controlled trial among 16 seasoned developers. Between February and June, the developers completed 246 tasks on popular open-source repositories that they were regular contributors to, such as the Mirror of the Glasgow Haskell Compiler and Hugging Face Transformers library. Participants were able to use any AI tool of choice and were compensated for their time.
Researchers recorded the screens of participants as they completed their assigned tasks. They then used the screen captures, as well as “rich statistics from source-code management systems” and interviews and surveys with participants to better understand the results. Some of the factors researchers attributed to the longer work time included participants overusing AI tools, believing they would improve productivity; existing familiarity with repositories, rendering AI tools less useful; and unsatisfactory outputs, leading participants to spend extra time trying different prompts or tinkering with AI-generated code.
Prior to starting the study, the developers predicted that AI would speed up their completion time by 24%. Their confidence in how much the tools would reduce their workflow fell slightly (20%) after participating in the study.
Grain of salt. The researchers of the study state that the factors they alleged contributed to the discovered slowdown are study-specific, and it should not be assumed that AI tools do not have practical real-world applications.
Top insights for IT pros
From cybersecurity and big data to cloud computing, IT Brew covers the latest trends shaping business tech in our 4x weekly newsletter, virtual events with industry experts, and digital guides.
“These results do not imply that current AI systems are not useful in many realistic, economically relevant settings,” the study’s authors wrote. “Furthermore, these results do not imply that future models will not speed up developers in this exact setting.”
Firsthand experience. METR’s findings come as “vibe coding” continues to gain popularity in the industry and developers embrace generative AI and LLMs in their workflows. A 2024 Stack Overflow survey found that more than three-fourths (76%) of developers used or planned to use AI tools in the development process last year.
Michael Armstrong, CTO at Authenticx, told IT Brew that the nonprofit’s discovery on AI tools tacking on time for some developers was unsurprising. He said he has personally seen occasional productivity losses when AI tools are introduced due to a variety of reasons, including learning curves, substandard quality, and distractions created by these tools.
“What I’ve observed is you can have a really, really good coder, but if they don’t really understand what they’re trying to solve for…the results are not ideal,” Armstrong said.
Meanwhile, David Murray, CEO of talent platform Confirm, said the study validated what he and his team had experienced in their own workflow slowdowns, citing hallucinations and developers tinkering with AI-generated code. He said it has influenced how he advises his team to work with AI tools.
“The guidance that we’ve kind of given each other is basically [to] be mindful and give very specific instructions and focus on specific use cases.”