What AI can and cannot do with a load test

When software engineers hand some of their day-to-day load to AI tools, they might include their load test, a common and important application check.

A load test reviews how a system behaves when subjected to a specific amount of user traffic or workload. Quality assurance (QA) engineers and developers define expected traffic and unexpected edge cases; testers then run scripts to simulate actions like logging in, searching, or purchasing—and see what lags, loads, and crashes.

Pros who spoke with IT Brew see AI tools helping with tedious aspects of the process, like finding testing cases and analyzing telemetry, but don’t envision those tools helping to run the test itself anytime soon.

A large language model is too unpredictable to be a “fire and forget” tool for load tests, Kevin McDonald, principal consultant at tech consultancy Hylaine, told us. “You have to have somebody there to monitor it. You have to have somebody there to make sure it’s doing what it should be doing, it’s testing the right things, and it didn’t go down a rabbit hole, or it didn’t start pulling in systems that don’t need to be involved.”

Where AI might pass the test. While today’s pros might not be ready for an LLM to provide the actual load test, some see AI removing some of the day-to-day burden from QA engineers:

• Generating test cases (and test scripts). McDonald envisions a helpful scenario where a developer offers an LLM a short app description and high-level architecture, and the model proposes common problems, test cases, and peak-traffic and normal load scenarios. Sai Joshitha Kathari, senior site reliability engineer at Visa (who spoke with us earlier about her first IT job), similarly sees an LLM potentially suggesting initial load-testing scripts, following a description of an API spec or application flow.
• Generating data sets for those test cases. Load test data must represent real production data; if a site usually processes twenty-page documents, one pagers won’t lead to an accurate test. In the past, a QA engineer may have spent time combing through databases and generating accurate test-sample data, according to McDonald. “The generative abilities of AI are being leveraged to generate those data sets very quickly,” McDonald said.
• Analysis: Load test results contain lots of telemetry across application logs, infrastructure metrics, and databases. Instead of manually reviewing every signal, AI can help summarize patterns worth investigating, according to Kathari. For example, a model might highlight that latency increased at the same time that CPU saturation occurred, or a downstream dependency started returning more errors. “I think one of the more useful applications is helping engineers move from, ‘The system slowed down,’ to, ‘Here are the components and signals you should investigate first,’” she wrote in an email to IT Brew.

Where AI might not pass the test. Kathari and McDonald both see limitations with AI-assisted load testing. The models can identify correlations without identifying the true cause, and may not know your context (like architecture, traffic patterns, and business requirements), Kathari noted.

Also, whenever you’re sending production logs and customer information to LLMs, that means cybersecurity and privacy controls must be considered. “I see AI as something that can help engineers investigate faster, but its conclusions still need to be validated against real telemetry and engineering judgment,” Kathari wrote.

Though Kathari sees AI as a true “assistant” helping engineers generate tests, explore possible failure scenarios, summarize telemetry, and form guesses regarding why performance degraded, she would not want an AI model to be the final authority on whether a system passed a load test, she wrote.

What to do. As for as how to exactly use the AI in a load-test context, Kathari recommends that engineers:

• Start small. Engineers should use AI for tasks requiring lots of manual time.
• Keep the actual performance thresholds deterministic. “For example, your system either stayed below the required latency threshold or it did not; the AI should not redefine that result,” Kathari wrote.
• Require engineers to validate AI-generated test scripts and recommendations before using them, particularly when the recommendations could affect production systems.
• Give the AI good context, including an understanding of architecture, expected traffic pattern, service-level objectives, and relevant telemetry. “Let AI help you investigate and reason faster, but let observable system behavior determine whether the test passed.”