Document, document, document: How to keep patching going when the patch pro leaves

While a VMware hypervisor is a complex, hardware-partitioning technology (and not a prop from VR Troopers), the reasons for leaving it unpatched are often frustratingly simple, according to Dan Wiley, chief security advisor at the security provider Check Point Software Technologies: There’s no one there to patch it.

One week after announcements that “ESXiArgs” ransomware was targeting VMware ESXi servers, telemetry from the infosec company Rapid7 found over 18,000 vulnerable internet-facing ESXi servers.

During Check Point’s February CPX 360 event in New York, Wiley revealed one example of forgotten fundamentals: “It is so basic…Why was that still open? And the reason we find is that the person that was responsible for VMware was gone two years ago, and they just didn’t know,” said Wiley.

Transferring patch information can be an organizational challenge, especially with technology like VMware that requires specific expertise. As IT teams add VMware servers to a list of assets that must be updated, pros are adopting tools and processes to ensure patch-management continuity.

“Big corporations, big banks know how to do this really well. They have a really tight life cycle management around it, but small to medium [businesses] struggle around this all day long as they just have the attrition and they don’t have the knowledge transfer from one person to another as things transition,” Wiley told IT Brew.

Tip the server. Companies reduce their need for hardware by running multiple services (Exchange, DNS, file-sharing, etc.) on the VMware server.

Between 2021 and 2022, the cybersecurity company Recorded Future showed about a triple increase in ransomware attackers targeting VMware’s ESXi servers.

This February, agencies, including CISA, noted that ransomware actors were exploiting known vulnerabilities in VMware ESXi servers that were likely running unpatched software. While VMware patched the flaw in February 2021, the attacks continue.

VMwhere is everybody? IT pros have gotten used to the routine of Microsoft’s Patch Tuesday—a monthly release of fixes that can literally go on a calendar. Though VMware makes regular patches, there’s no guarantee that an organization has a resident expert to handle them.

“You need to understand how the redundancies work, how to take the machines down, how to create backups off of them…And that’s a skill that not every sysadmin automatically has,” said Johannes Ullrich, dean of research at the SANS Technology Institute.

Configuration management databases (CMDBs) from vendors like ServiceNow and Device42, can note patch criticality, hosted applications, and owners of specific systems—valuable tools when a patch pro is absent. Other helpful documentation, especially during a transition, may include how to apply the patch, along with notes on dependencies and past issues.

“Part of the offboarding of a user should be to look at any systems or applications that they own, and reassign that ownership to either their manager or whoever’s backfilling their spot,” said Randy Watkins, CTO at the cybersecurity company Critical Start.

And if weekly, even daily reports on the state of applications, servers, and desktops aren’t happening, start those now, said Wiley.

“Someone’s going to leave eventually,” Wiley told IT Brew.