When app deployments go wrong—and how to get them right

A decade ago, one wrong drag-and-drop could lead to an IT version of the dreaded “reply all” nightmare, except instead of accidently sending everyone a photo of your dog, an engineer could deliver an unasked-for application to every company workstation.

With Microsoft Systems Management Service (SMS), now known as Microsoft Endpoint Configuration Manager (MECM), an admin could move one install package from the left side of the screen to a corresponding collection of target machines on the right.

And with a trip over to a folder called “All Systems,” an app meant for a few machines could go to, say, 45,000.

That was the scenario for enterprise IT professional Tim Flower, when he led a desktop-engineering team at a financial-services company more than 10 years ago. An engineer meant to send an Oracle database client to 2,000 employees, but after placing the production-app in All Systems, the install started going right to, you guessed it: everyone, everywhere, all at once.

“It was targeted to go to 2,000. At the end of the day, I think we were able to get it to stop after hitting maybe seven or eight thousand. But the potential was multiple dozens of thousands of devices. You get that pit in your stomach that you’ve realized that you could have just broken the company,” said Flower, who is now the VP of sales enablement at the IT services firm Nexthink.

How can an update break a company? The update could conflict with another program on the machine, for one. A forced reboot may also lead to a number of help desk calls, or angry employees who received the reset during a call with a customer.

“There’s lots of implications even to just a simple reboot, when you’re not expecting it,” said Flower.

The IT horror story—not a unique one, according to Flower—highlights the importance of a few app-rollout tools and strategies.

• Lead time: A rollout should be scheduled at least a week in advance, according to Flower. “You can’t come in and do a change control for tonight. You’ve got to have your test plan documented,” he said. 
• The test plan: Common strategies include running the update on virtual machines and watching for breaks (like Roman Shain from Nero Consulting previously told IT Brew), planning rollback strategy when conflicts arise, and post-deployment monitoring and logging.
• Automation: Research from Gartner in July 2022 predicted that half of enterprises will have implemented an AI-augmented testing strategy by 2027. Automated tools can take over tedious tasks that lead to human error. “We do poorly with boring tasks. And manual testing is like that,” said Melinda-Carol Ballou, research director of the application life-cycle management (ALM) program at the market intelligence firm IDC.

While Ballou sees disruptive app-deployment scenarios as rare—and Microsoft supports simulated and phased rollouts to avoid deploy-all scenarios—such disasters do lead to positive change.

“It’s those kinds of near-misses and those events that get people to change their behavior,” said Ballou.—BH

Do you work in IT or have information about your IT department you want to share? Email [email protected] or DM @BillyHurls on Twitter.