Why software developers need to watch out for package hallucinations

When in doubt, remain silent…unless you are an LLM, in which case, make something up!

Researchers from the University of Texas at San Antonio (UTSA) found that the above scenario is actually a somewhat frequent occurrence when developers use LLMs to generate code in a phenomenon known as package hallucinations, which occurs when models recommend a third-party software that is nonexistent.

The research team, which included contributors from the University of Oklahoma and Virginia Tech, examined 16 code-generating LLMs and generated a total of 576,000 code samples in Python and JavaScript for the study. They found that on average, 5.2% of the packages generated by commercial LLM models were hallucinated. For open-source LLM models, this figure jumped to 21.7%.

To make matters worse, the researchers discovered that hallucinations produced by the LLMs aren’t usually “random errors.” Using a sample of 500 prompts that generated package hallucinations, the team reran their initial queries 10 times and found that 43% of the hallucinated packages were repeated in every attempt.

Bad package. Joe Spracklen, a second year PhD Student at UTSA and lead researcher of the study, told IT Brew that the research shows just how pervasive the issue of package hallucinations is for software developers.

“This is a phenomenon that occurs across all models and at a fairly large rate,” he said.

Spracklen said threat actors can leverage package hallucinations to perform a package confusion attack, where they publish a malicious package with the same name as a hallucinated package to code repositories. He described the scheme as a “modern day typosquatting attack.”

“There are no confirmed package hallucination attacks in the wild specifically,” Spracklen said. “Although it’s almost certain that people are trying to do this.”

In their LLM era. The findings come at a time where developers are increasingly embracing more AI tools in their workflow. Stack Overflow’s annual developer survey found that 76% of queried developers claimed last year that they were already using or planned to use AI tools in their development process. Spracklen told IT Brew that an increasing trust toward LLMs can increase a developer’s vulnerability to package hallucination-related attacks.

“As we become more trusting of the models, we become more susceptible to just blindly trusting their output and then potentially blindly downloading package code that we have not vetted for ourselves and have not done our due diligence on,” Spracklen said.