When a test platform fails

If you’re not careful, a test environment can turn into a live-run of a security disaster.

When Rob Forbes was working as a senior security architect, just a few years ago, he watched a major retailer inadvertently use live customer data in a platform not meant for production. The test platform—a customer identity and access management system, he recalled—was tied to production databases, so changes made in the former “rippled down into the production environment.”

“We had to quickly yank that back down before it got found and scanned on the internet,” Forbes, now field CISO at cybersecurity services company Stratascale, told IT Brew.

Forbes shared how his team revamped the change-management process following the rough test—a phase of software development that leaves plenty of opportunities for access and data leaks.

It’s just a phase! There are typically four stages of software development:

• A developer platform for early building
• A test platform—a complete setup of applications, infrastructure, and data that allows software to be tested for functionality and security
• A staging platform, which is meant to simulate production before moving to the actual…
• Production environment, which is live

The test platform is in a controlled, non-production setting, not meant for public access or to host production data, according to John Pettit, CTO at Google Premier partner and consultancy Promevo.

“People start to blur the line between the test and staging,” Pettit said.

DevOps company Perforce Software, in a global survey of 250 orgs, found that 86% of organizations allow “data compliance exceptions” in non-production environments.

Driving factors. With test environments, Pettit sees a conflict between a team of fast-moving developers and a team of security practitioners who want some questions answered first.

“You have a culture responsible for protection, and you have a culture who’s responsible for building stuff and getting things done,” Pettit said. “The ‘getting things done’ team once they’re given access to something, rarely has the security mindset to think about, ‘How do I keep us safe?’”

Forbes added that access and data shortcomings often happen in test platforms because developers may be unfamiliar with a given cloud platform’s controls.

Let’s review. Following the incident with the retailer, Forbes said his team amassed “an inventory of failings” and tasked themselves with asking developers questions like: Why was production data used in test environments? Why were developers given admin accounts created for database access?

Following the incident (and for future test platforms), Forbes and his team created a process that snapshotted customer data and ran it through a data hygiene tool to scrub out sensitive information like account numbers and addresses.

Other companies rely on similar methods to ensure their valuable data is safe in case of an incident. In July, for example, Dell confirmed a breach of its Solution Centers test environment—a way to preview features to customers—and told BleepingComputer that the data used “is primarily synthetic (fake) data, publicly available datasets used solely for product demonstration purposes or Dell scripts, systems data, non-sensitive information and testing outputs.” (Dell did not share the cause of the breach.)

This is only a test! Independent software vendors (ISVs) like Ensono sell test-server licenses to customers.

Once a “test” environment includes company data, Pettit treats the “test” like a production environment, recommending hardening practices such as password-rotation requirements, identity-aware proxies that restrict access over the internet, and monitoring from a security operations center (SOC) environment.

“I think that word ‘test’ is sort of a dangerous word to start with,” Pettit said. “If it’s a publicly accessible server that has your credentials, if it can connect to your data, connect your other internal systems, it’s not a test system.”

The next test. Forbes sees the test problem only becoming more challenging as companies try out AI.

A Q2 Gallup poll found that 44% of surveyed US employees said their orgs have implemented new AI tools—up from 33% in 2024.

Agents will need to be instructed on what coding languages, platforms, language models, and servers it can leverage, Forbes said, and it must be equipped with access restrictions and logging.

Agents, perhaps, are on the “getting things done” team.

“There are a lot more controls that we need to apply to AI because, again, it’s just trying to complete the task,” he said.