How data center pros handle a snow day

When a snowstorm barreled down the East Coast in late January, Jim Kozlowski’s data center team in North Carolina had a sleepover of sorts.

Fortified with stocked refrigerators and bedding, three to five technicians were ready to spend the next 48 hours checking generators, fuel lines, and other systems keeping the facility running.

If the data center went down, so would the servers and IT infrastructure of the large enterprises that Ensono supports all day, every day, even when it’s snowing hard.

“I can’t afford a second offline,” Kozlowski, Ensono’s chief sustainability officer and VP of data center operations, told IT Brew.

Data center pros spoke with us about the preparations they make during a snow day—as well as the days before the blizzard.

What’s in a data center? When a data center loses grid power—always a possibility during a storm—either diesel generators or other backup power will kick in. Nonetheless, it’s always a good idea to have people sleeping over to ensure those alternatives are working. A service-level agreement (SLA) may require data centers to have near-100% uptime, according to service tiers established by the Uptime Institute more than 20 years ago.

All-nighter! During the snowstorm, Kozlowski said, the sleepover team went on rounds every hour to inspect and fix any systems at risk of going offline.

For example, according to Kozlowski, a fuel line threatened to “gel over.” Restarting the generator got the mixture flowing again.

“You always want to get ahead of a problem,” Kozlowski said. “It is hard to recover from something that breaks or has an issue during the middle of a storm when it’s five degrees outside.”

More snow days. During a blizzard, if you dare go outside, you just might see a member of Joe Minarik’s team on a data-center roof. Minarik may ask members of his teams supporting 70-plus facilities around the country to clear snow and ice before it can wreck infrastructure.

During a recent snowstorm in Houston, Minarik, chief operating officer at DataBank, said technicians spotted and quickly sealed a leak in a data center caused by frozen pipes. Unspotted, those drips could turn into a “catastrophic failure.”

“That could be affecting a critical air conditioning unit,” Minarik told us. “Now the data center will overheat and then we’ll have to shut that equipment down.”

DataBank technicians sometimes have to roll out the cots for a blizzard. There’s also a “sister site” well away from the storm where fellow engineers can remotely detect unusual activity like unexpected temperatures (or leaks!).

Snow day readiness involves regular preparations on sunny days, too. Throughout the year, Kozlowski said his team routinely double-checks generator fuel levels, tests batteries, communicates with fuel vendors, and determines which customers might be planning major IT changes during a blizzard (and perhaps convincing them to delay deployment to when the shoveling’s done).

“It’s still a very labor-, physical-oriented job for people. We need people,” Minarik said. You don’t have to worry about AI taking your job here.”