Q&A: How AI has ‘radically changed’ the role of data engineer

If you see a data engineer these days, give ’em a pat on the back. They’ve been busy.

Behind every AI chatbot drafting an email or answering your recipe question, there’s likely a data engineer doing the data equivalent of kitchen prep.

A data engineer gets information ready for processing, which includes deleting duplicates, anticipating queries, and anonymizing information to meet compliance standards. All of those tasks, thanks to AI, are multiplying.

A recent report from MIT Insights, produced in partnership with data and AI platform Snowflake, polled 400 CIOs, CTOs, and other tech-minded business leaders. The study revealed an increased role—and workload—for data engineers:

• 81% of executives said the data engineer job description has “changed radically due to AI.”
• 77% of respondents said data engineers’ workloads are getting heavier.
• 55% of CIOs thought data engineers are “integral to the business,” which is low compared to the 80% of chief data officers and 82% of chief AI officers who find the role essential.

Chris Child, Snowflake’s VP of product and data engineering, has seen the evolution of the data engineer role firsthand. Snowflake has its own internal chatbot for business users; the platform answers questions by pulling from structured and unstructured data related to customers and product usage. The engineers have to get that data “in shape,” Child said, so that the tool can answer questions about customer activity.

According to Child, that means encoding “semantic models,” or context, into data structures so AI agents can understand business questions. The definition of a “customer,” for example, differs from company to company, and an engineer must anticipate query types and make sure relevant data reaches requesters in proper formats.

“Setting up the semantic model and giving examples to help the agent be able to answer the question appropriately: That’s work that the data engineering team is spending, I think, more time on than they had in the past,” Child told us.

We spoke more with Child about how data engineers are spending their time.

Responses below have been edited for length and clarity.

Executives say the job description for data engineers has changed radically due to AI. What’s one example of a radical change?

The volume of data that most companies are having to collect and deal with has grown exponentially…With AI, you’re starting to be able to get value out of unstructured data, get value out of broader sets of data that were hard for people to think and reason about in a reasonable period of time.

How else has the role changed radically?

It used to be the data engineer’s job was mostly to get data ready for data analysts; they were really who their end customer was. But we’re now seeing a lot of [agentic] AI able to tap into and start using this data. And so the number of consumers of the data is also growing quite a bit.

How is the workload getting heavier?

The number of these agents that you can start deploying is significantly higher than the number of data analysts you’re planning to hire. So, it used to be if you wanted to scale the number of people asking questions, you had to hire a bunch of data analysts. And it’s relatively easy to figure out some ratio of data analyst to data engineer: You kind of hire both at the same pace. With AI, you’re able to suddenly 10x or 100x the number of requests coming in, and that leads to just a growth in [the need for] data structure and new data sets…Data engineering still seems to be harder to automate with the LLMs as they exist today, although I think that will change over time, but you end up with a disconnect where you need more work to be done to support the big number of consumers of that data that’s growing quickly.

Where does the data engineer shine now? What’s an important skill set?

If you go back and you look at some early data engineers, those people were writing scripts, and they were running them from their laptops, and then figuring out how to get a server up somewhere that was going to run and manage it. And there was a lot of infrastructure that you needed to manage…As [data engineers] are starting to use more AI to write transformations or using AI within the way that they’re operating, they’re going to be able to think about, “What are the business outcomes that we want to drive with this data?” That’s going to allow them to be thinking much more, almost at a strategic level, as opposed to just the tactical—how am I transforming from this piece of data to this piece of data—and actually become more critical partners to the business.