How to develop an effective IT disaster recovery plan

If your organization experienced a ransomware attack today, how long would it take to recover?

That answer is likely dependent on your IT disaster recovery (DR) plan, a formal document that details how an organization will restore IT infrastructure in the event of a disaster. In this context, such a catastrophe can be anything “where the technology interrupts the operations of the business,” according to Michael Sparks, principal cloud operations engineer at Nextworld.

“That could be something as simple as somebody trips over a cable in a data center,” Sparks said. “In large organizations, that could be a hurricane is bearing down on the East Coast, and a data center is about to get impacted.”

Sparks said the ultimate goal of the DR plan is to make sure that a business and the technology supporting it can continue to operate after a disruption. IT Brew caught up with several IT pros to understand how to craft an effective plan.

It’s pretty important. There are many reasons why a business should prioritize having a solid DR plan. Phil Conway, developer advocate at SandboxAQ, told IT Brew prolonged downtime could prove costly for an organization, and can even inflict reputational damage.

“It’s not just reputational and cost risk, it’s the risk of that service not being available in the desired state [or] not being brought up securely or quickly,” Conway said. “If you don’t bring [a core foundational service] up in the right configuration, there might be performance issues. You may find that everything runs really slowly [and] your customers complain.”

Kasey Best, VP of threat research at DNSFilter, added companies without a strong DR plan also risk being unable to recover quickly during a time of need.

“There’s just a whole bunch of risk that you open yourself up to when all it would have taken was some preplanning,” Best said. “The more crucial your services are and the more all of your services are interconnected or interdependent on each other…the more important disaster recovery is.”

Best practices. Before crafting a DR plan, Conway said organizations need to have an understanding of their recovery time objective (RTO), or the maximum amount of time a system can be down before it starts to disrupt a business, and recovery point objective (RPO), or the maximum amount of data loss that can happen during a disaster without impacting the business.

“Businesses need to think: ‘How quickly do I need to be back up on my feet? What’s the cost to my business if I’m not available? What’s the reputational damage if my services aren’t available?’” Conway said.

Best added that companies should define their scope of infrastructure and data, and pinpoint the types of disasters they are planning for.

“Figure out what you’re planning for, identify what depends on that being up, identify if you can back up, and then measure how long it takes you to get back to full operating capacity,” he said. After constructing the DR plan, Best suggested sharing it with tech stakeholders on the frontlines to get additional feedback on whether the strategy is sufficient enough if something goes wrong.

“You won’t get the same answer you will asking the people on the ground as you will asking their managers typically,” Best said. “I would suggest that you do both. You run the plan past them, and then you say, ‘Alright, this is what you all told me. Now we’re doing a live fire exercise to see if you’re right, and see if the plan changes.’”

Companies should also have a good understanding of what their full operating capacity might look like post-disaster under a DR plan.

“If you’ve got 10,000 site visitors, can your backup version handle 10,000 site visitors every hour? Can it only handle 200?” Best said. “When your stuff goes down, a lot of people will be sitting there trying to refresh and get everything up and working. When you come back online, you’re probably going to have more traffic than you normally do. Can you handle that?”

Testing, testing 1, 2, 3. After a DR plan is formed, Conway said regular testing is needed to ensure business operations can actually continue in the event of an emergency.

“You’ve heard of Schrödinger’s cat. There is also Schrödinger’s backup, which is a backup [that’s] neither working nor non-working until you actually try to recover all of the data that’s in it,” Conway said. “The same is true for DR. Unless you test your plan regularly and review it regularly, it’s a potluck as to whether or not it will work when you actually try it for real.”

Sparks said organizations should practice and get comfortable with a full failover.

“Figure out how to really, really run your business processes out of whatever that failover state is because until you do that, you’re not going to discover all the systems that people depend on,” Sparks said.

Revisions. Finally, Conway said DR plans are “living, breathing documents” that should be continuously updated as needed: “As you adopt new applications, as you open new offices, as the needs of your business changes…you should be looking at the disaster recovery plan and updating it accordingly.”