When an end-of-the-workweek CrowdStrike update temporarily turned some Windows machines into blue screens of death, many organizations (and universities) had to scrap their Saturday beach plans, head to the command line, and reboot a bunch of computers.
R. Hal Baker, a doctor, SVP, and chief digital and information officer, and his IT team at WellSpan Health, an org with 9 hospitals and 250-plus patient locations across Pennsylvania and Northern Maryland, had to address almost 3,000 devices. That meant about 70 people volunteering over the weekend to provide hands-on-keyboard remediation.
“It almost looked like the flu-shot line or Covid-shot line, except people had their laptops and they were getting fixed,” Baker said.
With a Friday outage, according to Baker, some systems—thankfully, like those in the billing department—could wait until Monday. Regarding weekend remediation plans: “There was a certain opportunity to move non-disruptively,” he said.
Baker and other IT pros who spoke with IT Brew considered what they’d prefer: A Tuesday IT emergency (when everyone’s online and easy to find) or a Friday one (when employees have clocked out and IT pros have off-hours time to investigate).
Everybody’s troubleshooting for the weekend. Plenty of IT emergencies happen as the weekend closes.
- A late 2021 flaw found in the logging utility Log4j, led one interviewee to tell the Department of Homeland Security (DHS)’s Cyber Safety Review Board: “No one in the industry was sleeping that weekend [following the vulnerability’s announcement]—they were trying to patch millions of servers.”
- A report from cybersecurity firm Sophos said 43% of ransomware attacks in the first half of 2023 occurred on Friday and Saturday.
- CrowdStrike’s configuration update impacted hosts that “were online between Friday, July 19, 2024 04:09 UTC and Friday, July 19, 2024 05:27 UTC,” according to company details.
Stephen Watt, SVP and chief information officer at enterprise-software provider Hyland, remembers a cyber-incident that blurred into the weekend—and his Hawaii vacation.
Top insights for IT pros
From cybersecurity and big data to cloud computing, IT Brew covers the latest trends shaping business tech in our 4x weekly newsletter, virtual events with industry experts, and digital guides.
“It was probably Friday morning, very early there, and my wife was asleep, my family’s all asleep. And…I had some messages on my phone and some responses around what ended up being a huge systems-related outage and response situation that we had to deal with,” Watt told IT Brew.
Count Watt as someone who appreciates a weekday emergency, or at least prefers a Tuesday incident to a Saturday one. For one, “most people probably didn’t have any plans on Wednesday, other than to go to work,” Watt said, adding that business leaders are easier to find on a weekday.
As a former IT admin and network engineer, Vadim Vladimirskiy, current CEO of cloud-management provider Nerdio, prefers the solitude of a weekend IT problem. "End-users create more stress and angst for the IT admins than the systems themselves. So, having users who can just say, ‘Well, look, I can just start my weekend early, and you have two days now to get me back up and running by Monday morning, I think, provides a certain level of calm to the IT administrator,” Vladimirskiy said.
A disruptive weekend CrowdStrike update, in a way, was business as usual for Baker’s staff, a team that frequently operates during quieter times.
“Most IT departments in healthcare work off hours. So, if we need to do an upgrade, we don’t do it at 12 noon on a Wednesday. We do it at 2:30 am on a Sunday, because it’s least likely to impact the health system at that point,” Baker said.
In an emailed statement to IT Brew, CrowdStrike’s senior director of corporate communications Kevin Benacci wrote: “CrowdStrike’s focus continues to be on using the lessons learned from this incident to better serve our customers and we have outlined in our RCA the steps we are taking to help prevent this incident from happening again.”