How AI can ease runbook writing

Nobody likes writing a runbook, according to Todd Thorsen, CISO at data resilience and protection platform CrashPlan.

While runbooks’ standardized instructions for go-to IT problems can prove helpful for both new hires and on-call team members, producing these guides is “extra added overhead,” Thorsen said.

An IT pro has enough going on without spending hours laying out the directions for hardening a Ubuntu server or troubleshooting API latency with a payment platform.

But that extra workload hasn’t stopped Thorsen’s team from writing about 20 runbooks for their organization, including one on how to deploy vulnerability sensors in a new cloud environment.

Runbooks are important ways to pass on knowledge gained through experience, Srinivas Chippagiri, a senior member of technical staff at Salesforce, told IT Brew. His team has about 10 runbooks covering topics like onboarding a new engineer to a platform or restoring a failed service.

The internal guidebooks may feature experiences “that only engineers who worked on the product know about that may not necessarily be covered from manuals,” Chippagiri said.

But writing—and updating them—is a lot of work, and developers and IT leaders like Chippagiri and Thorsen have turned to AI to take on at least some of the burden.

Let’s get started. Runbook creation often involves time-intensive processes: narrowing in on a specific problem, then writing the (often many) steps to truly resolve it in a style that connects with a variety of intended users, from intern to IT veteran. A runbook author might also need to understand and note the severity of certain alerts, business impacts, and dependencies on other services, and know important contact info for whoever can solve a specific problem.

As with many tedious tasks that nobody likes doing today, IT pros have a chance to hand runbook creation to a large language model (LLM) and see what comes back. But for the output to be helpful, Thorsen recommends a solid amount of input, including network diagrams, cloud configurations, and other procedure documents with context regarding alerts and previous incidents, disaster recovery steps, and cloud-environment best practices.

A prompt to his LLM of choice might be: Using these documents as a reference…create a runbook to reconstitute an AWS environment.

“It’s important, obviously, to have a subject matter expert who is doing the prompting and then doing the review of the output,” he said.

From there, a reviewer fills in the gaps. Thorsen, for example, makes sure his runbooks are prescriptive and contain screenshots that articulate visually how to accomplish a task.

For Chippagiri, AI has been a documentation “game-changer,” helping teams quickly provide a runbook that’s about 60% of the way there, following an input with design docs, project management artifacts, and thoughtful prompts. Human writers, he added, are left filling in gaps that LLMs might not figure out, including new permissions, outdated links, or judgment calls that only a seasoned employee would understand.

Update! Thorsen has both formal review processes—annual assessments, for example, regarding what documentation needs updating and what changes have occurred in the IT environment—and informal ones. If a CrashPlan team member is working with a runbook and sees outdated information, they are encouraged to immediately provide the update and notify the team (via the Jira service-ticket platform) of any new details.

Chippagiri has even used AI for the runbook updates. “We can tell the AI to go and edit certain portions of the runbook,” he said. “You just have to do some babysitting, and then some fact-checking, and also you have to make sure that the procedures are actually correct once AI has done it.”