How to find the right amount of hardware (or software) for the AI job

Some AI jobs demand a huge hardware buildout in the office, others require nothing more than a subscription to a cloud-based AI service such as OpenAI or Google Gemini.

Deciding whether to build out a customized AI platform, buy (or rent) one from a third-party vendor, or wait to see what the future brings is a critical choice, especially as many AI deployments haven’t paid off yet. For IT pros who decide to build, how can they ensure their AI hardware and software doesn’t become “shelfware”?

A 2025 MIT study revealed that 95% of 300 publicly disclosed AI initiatives found zero return on their investments.

In many cases when building out customized AI, “the reality is the servers become the equivalent of shelfware,” Dean Chyla, solutions director for compute and AI at solutions integrator Insight Enterprises’s Infrastructure Center of Excellence, told IT Brew. “They’re sitting there. Maybe configured, maybe not configured, certainly not optimized. And there’s kind of a long road for customers like that, where they need to build the skills to actually use it.”

If you build it. An org may decide to build their own large language model, or what Sundeep Goel, CEO at financial-control platform for AI infrastructure Mavvrik, called a “mini-AWS” (referring to Amazon Web Services). With the help of hardware like servers and private GPU clusters, companies can create the infrastructure they need to run an AI workload.

Goel sees cost and security as driving factors for this “repatriation,” or the returning of AI services from the cloud to a private data center, especially for industries that face regulations and need tighter control and visibility of their data.

“There’s a real sensitivity around sharing data in a public cloud where your competitors may benefit from it. Or you’re in a regulated industry, and you need to be very careful about what data you use to train public LLMs,” Goel told us.

A Mavvrik study found that 67% of 372 global companies are actively planning to repatriate some AI workloads to owned infrastructure, and another 19% are evaluating the move. “Companies that charge for AI are more likely to plan repatriation, linking monetization with tighter control over infrastructure,” the report said.

But wait, there’s more (cost). While hardware implementations could include servers and GPUs to allow customers to run multiple workloads, the investment doesn’t end there, Chyla warned; there is also infrastructure required, like high-speed networking and storage to support the GPU-enabled servers and feed data into the GPUs. Setting up and maintaining this infrastructure requires IT pros with considerable expertise.

The way to avoid a lack of return on ROI, Chyla advised, is to start with a good plan that engages organizational leaders.

“This should not be something that’s just pushed down from leadership without any input from stakeholders in the business,” he said.

Before buying hardware, Chyla recommends business leaders ask questions like:

• What are the use cases that will impact your business? What business units have generated those use cases?
• What platform have you chosen to run this on? (“If you choose a pure cloud provider, like one of the hyperscalers, then you’re locking yourself into using that platform for everything. So, is that acceptable to the business?”)
• What are your regulatory and security requirements?
• Do you have a center of excellence? Who are the stakeholders in that center of excellence?
• How will you show ROI?

Goel advised clients, especially software-as-a-service companies deploying their own AI products and training their own models in-house, to do financial analysis before making long-term infrastructure decisions: Measure total costs of ownership in a small pilot-level, and then examine costs at scale.

Big AI on campus. Some research projects at the Grainger College of Engineering at the University of Illinois Urbana-Champaign require GPUs. Professors, for example, may need to create specialized models for a task using their own research data—say, images of weeds for agricultural recognition systems—to existing models. That fine-tuning requires hardware.

Other Grainger deployments, like the school’s CropWizard from UI U-C’s Center for Digital Agriculture, can use commercial models like ChatGPT or Gemini to consult a specific set of documents for specific answers to agricultural queries. The method, known as retrieval-augmented generation, customizes the general purpose model and requires no GPUs or special hardware.

“By doing that and then tailoring the software stack on top of them for a particular product use case, you can do many, many things today in the AI space without having to train your own models at all, and without even having to host the model,” Grainger computer science professor Vikram Adve told IT Brew.

MIT’s report found that tools like ChatGPT and Copilot are “widely adopted.” Over 80%of organizations have explored or piloted them, but these tools “primarily enhance individual productivity,” not profits and losses. A further 60% of orgs evaluated enterprise-grade systems, the report revealed, but only 20% reached pilot stage and just 5 percent reached production.

Chyla sees the failure as “misalignment with the business.”

“IT is catching a lot of pressure from board of directors, lines of business, executive management to say, ‘We need to do something about AI, without necessarily the support of having an AI strategy in place, [or] having clear use cases defined from business units,” Chyla said.