How one data engineer got out of the silo (and how you can too)

In 2018, Justin Goff, then a solutions architect at a medical equipment distributor, often sat at his desk working away on a SQL database, maybe with one standup call on the calendar.

When Goff learned that his company had hired a firm to redo the UX of its e-commerce site, he thought the planners had underestimated the project’s required data-migration tasks, including how back-end systems functioned in a revamped, multi-site context. And he saw an opportunity.

“I made this massive project plan draft and outlined the full end-to-end spectrum on how I thought it should go,” Goff told IT Brew of his phased strategy, which included cloning three legacy databases in a staging environment. “I presented it to my project manager and the director of marketing at the time in a meeting with the dev team there as well.”

Goff would end up presenting his plan to the CEO and other leaders, complete with printouts and language that he felt was tailored to the executives.

Recognizing—and acting upon—a business-impacting problem can help data engineers and architects stand out. But they have to stand up from behind the desk to do it: speaking to senior leaders, embracing upcoming challenges, and finding strategic ways to solve them. If they succeed, it can give their careers a significant boost—and make them known throughout the company as more than just “one of the IT people.”

It’s in the name. Goff described two distinct data worlds at the medical equipment distributor where he worked at that time: e-commerce and internal operations.

“We often didn’t know what the other hand was doing,” Goff said of his stint on the e-commerce side. Additionally, he said, the intricacies of following database connections can be so thought-consuming that “you just naturally kind of stay heads-down.”

Goff said that, as a database administrator and architect, he often felt like he lacked an understanding of the business context behind his work. And he’s not alone: For data engineers and other IT pros, it can often prove difficult to find the time and opportunity to collaborate deeply with other business units. AI tools, too, have reportedly made the job less social: a recent report from MIT Technology Insights (produced in partnership with cloud-native data platform Snowflake) found that 77% of business leaders said their data engineers’ workloads are increasing, thanks to AI-focused efforts like managing unstructured information.

Let’s meet up. Throughout his career, Diptamay Sanyal, principal engineer at cybersecurity company CrowdStrike, has found himself in weekly “syncs”—meetings with product leaders, data engineers, and sometimes a marketing team member or business stakeholder to provide updates on new features or ideas.

For a data architect to feel like a partner to the business, he said, it’s important for them to understand business and product problems before writing any code. Sanyal has done this in a variety of ways throughout his career, including asking for and analyzing tickets from the issue-tracking platform JIRA, to discover pain points. From there, he’ll “work backwards” to figure out data-specific solutions.

“It depends on the entrepreneurship of the data engineer at the end of the day,” Sanyal said.

Entrepreneur mode. For the e-commerce UX project, Goff needed a week to develop his full plan: cloning three legacy databases into a staging environment; merging those copies into a single multisite database; migrating that to a second database, following important schema changes; applying configurations from the new site; building an indexing layer to track changes and new rows; and doing all that while keeping the site up and running.

His first presentation (to his developer team) involved a massive, granular spreadsheet. When it came time to speak to the execs, however, he geared the presentation toward milestones, phases, and business value—like highlighting how a successful data migration could boost the CEO’s revenue goals.

“When I presented the project plan, there was absolute surprise from the business side, because they did not expect that…I was going to do that. But they also didn’t even know that I had this expertise, because I was pretty new at the company,” Goff said.

And the migration was largely a success, he told us—so much so that he didn’t need to administer the new database.

Goff said he went from being “a bridge between DevOps and transactional databases” to a more specialized role in data science—one that involved modernizing the company’s analytics. Goff, now director of technical delivery at tech consulting firm Hylaine, sees curiosity (about the “why” behind a project) as an important driver for today’s data engineers and architects.

Data engineers can help business leaders determine if emerging technologies like agentic AI have proper workflows and governance, for example, and if those decisions are actually making the company money, Goff told us.

That kind of conversation can take a data pro outside the silo, into executive conversations, and perhaps into a new role altogether.

“Are there opportunities for me to interject my opinion, where it could be valuable from a business perspective?” Goff said. “Those are the moments that you may come across where you could make a career-defining change.”