Can on-prem solutions beat cloud in the age of AI?

AI solutions are cloudy with a chance of prem.

Third-party, cloud-based AI vendors are becoming cost-prohibitive, so companies are turning to developing their own AI tooling on internal tech stacks, backed by local hardware.

Running your enterprise’s AI solutions on-prem was long reserved for the big dogs of Silicon Valley, who could afford infrastructure and developers. But a growing number of free tools allow smaller companies to deploy locally, changing the game.

A February study from Lenovo found that on-prem AI infrastructure is more cost-effective, especially given the rising cost of tokens: “For enterprises committed to AI as a core competitive advantage, the transition from renting intelligence to owning the factory is not just a technical evolution, it is a financial imperative.”

Part of the reason for that, said Lenovo Global Head of AI Robert Daigle, is that democratization of AI has led to infrastructure and performance improvements, leading to an acceleration in capability and efficiency. That makes it smoother for organizations to leverage AI, Daigle said, and for hybrid solutions that combine on-prem and cloud.

“Evaluating the cost difference between using a cloud-based system and using a private hybrid AI platform to run your AI workloads, and just comparing the cost, we’re seeing now that you could have a simple payback in under four months for on-prem infrastructure, which is pretty impressive,” Daigle said.

Cash up. The rise of tokenization and the increasing costs of maintaining data flows make open-source AI solutions attractive. As Anil Nanduri, Intel’s VP of AI product management, told IT Brew, organizations can gain quite a bit by leveraging a cheaper, on-prem infrastructure—think open-source solutions run on local hardware—alongside existing offsite technology like tools provided by AWS or Anthropic.

“Customers who actually have on-prem infrastructure, they’ve always been kind of hybrid, they’re using hyperscalers for a certain amount of compute,” Nanduri said.

Control is also a concern, Chris Anley, chief scientist at NCC Group, told IT Brew. Enterprises need to have a sense of how to manage their models and keep them secure.

“If an organization has a requirement where a model must always be available, then hosting a slightly smaller version of that model locally on premises, or in their own cloud, so that they can control it and make sure that it’s always available, is very important,” Anley said. “There’s a sort of availability/sovereignty thing there; there’s also cost—CFOs are rapidly learning that embracing AI definitely comes with a big price tag.”

Eye on the ball. Ultimately, Nanduri told IT Brew, changes in technology will drive how the tech stack is managed. In that context, hybridization is more the natural evolution of internal processes. It also means more capital expenditure on the nuts and bolts of computing—CPUs.

“Any time automation comes in, machine to machine, you’re getting efficiencies, you’re going to be making things more productive,” Nanduri said. “When you start deploying more AI, you are going to have infrastructure that is needed that actually turns into different compute systems, like you will need more CPUs now to execute what the AI is generating.”