A look back at Log4j shows fast reaction, slow remediation

It’s perhaps no surprise that when somebody comes up with a great fix in life, few people actually use it—many a Scrub Daddy, Squatty Potty, and Rapid Ramen Cooker stay sealed and unopened, despite being upgrades to their predecessors.

In the security space, better versions of products are released all the time, at high speeds, and customers still need to be convinced to adopt them.

When a security engineer in late 2021 discovered a vulnerability in the open-source Java-based logging framework known as Log4j, the response was swift. A fix was up for review five days after the November 24 finding, and the Log4j upgrade was available by December 10.

That’s prompt patch-making, but a number of organizations have taken a slower approach to deploying the update. Log4j is integrated into millions of computer systems, including ones used by governments, but many companies still lack asset-management and patch-testing practices that remediate the security threats caused by outdated versions of Log4j—or any outdated software, for that matter.

“We’re not stuck on identification of a problem. We’re really stuck on configuration and change management, and then creating a process there for teams that allows them to do that in a reasonable timeframe,” said Rick McElroy, principal cybersecurity strategist at VMware.

Collaboration congratulation. As for identification of the problem, the Cybersecurity and Infrastructure Security Agency (CISA) praised early Log4j support efforts. In its July 2022 report, the department lauded vendors’ rapid advancement of threat information and the nonprofit Apache Software Foundation’s well-established software development cycle.

Yet organizations struggled to respond, said CISA, citing a slow response as companies weighed a classic IT debate: patch deployment vs. possible downtime. “The hard work of upgrading vulnerable software is far from complete across many organizations,” reads CISA’s research.

Open season. Vulnerable instances of Log4j are easy to find, even one year later. In April, a team from Rezilion discovered more than 90,000 internet-facing applications and more than 68,000 servers open to Log4J exploits. In November, CISA and the FBI announced that Iranian threat actors are still exploiting a Log4j vulnerability in unpatched VMware Horizon servers.

Companies with a lock on Log4j response, said McElroy, know their assets and have a path to remediation.

Software composition analysis tools help developers track the open-source components in an application. External attack surface management (EASM) products find external and internal-facing assets. A software bill of materials (SBOM)— a White House-recommended practice—provides a component inventory.

Security teams must connect with application developers about risks related to factors like who has access and if the service is publicly exposed, said Kristen Bell, director of application security engineering at the cybersec firm GuidePoint Security.

“You’re really reliant on a community to enhance and support those open-source components,” Bell told IT Brew.

Expect exploitation. Inventory aside, patch processes often require testing in a virtual, non-production environment, along with continuous monitoring—efforts that understaffed IT shops may not have the time for when it’s time to deal with Log4j.

“There are a large number of organizations out there that aren’t dealing with it effectively—lack of staff, lack of budget to do upgrades. The SMB [small and midsize business] market in particular is really vulnerable,” said Doug Saylors, partner in the cybersecurity practice at the consultancy ISG.

McElroy thinks Log4j exploits may continue, because outdated versions of things tend to stick around. (There are still over 4.5 million internet-facing devices are open to “vintage” CVEs discovered between 2010 to 2020.)

“It wouldn’t honestly shock me two years from now to hear somebody got exploited with Log4J,” said McElroy.—BH

Do you work in IT or have information about your IT department you want to share? Email [email protected] or DM @BillyHurls on Twitter.