Houston, we have a bot problem…
Web traffic to some webpages has started to resemble a Los Angeles freeway as AI bots, also known as crawlers, increasingly overload servers with requests. An April Imperva report found that automated traffic, including both “good” and “bad” bots, made up 51% of all web traffic last year. While this marks the first time bot traffic has surpassed human activity, WP Engine VP of Product Management Christine McKee told IT Brew that the influx of automated traffic feels familiar.
“I would say that it’s similar to when spam came into the world and email, where all of a sudden people started getting more and more emails, and it was slowing down the whole system,” she said.
For WP Engine, a hosting platform that hosts over 1.5 million WordPress websites across more than 150 countries, defending websites in its network against the unintended effects of high bot traffic is a top priority, according to McKee. McKee told IT Brew that the platform has seen a “definite rise” in the traffic experienced by its supported websites.
“Bot traffic affects performance. Bot traffic affects cost,” McKee, who joined WP Engine in 2023, said. “Bot traffic, basically, can be difficult for customers to understand and deal with.”
Bot busters! McKee told IT Brew that WP Engine recognizes that not all AI bots are necessarily bad, despite their negative reputation.
“Anyone that’s actually used any LLM capability, whether it’s a distinct one like Claude or ChatGPT or Copilot, any of those, they add a lot of value to your world,” McKee said. “So, the things that are powering them and discovering all of the content in the internet and then creating the models from that, they’re not bad.”
Top insights for IT pros
From cybersecurity and big data to cloud computing, IT Brew covers the latest trends shaping business tech in our 4x weekly newsletter, virtual events with industry experts, and digital guides.
Because of this, prioritization is a key part of how WP Engine addresses bot traffic.
“For our system, we prioritize humans over bots,” McKee said. “So, when we’re serving up traffic, we try to serve up the pages to humans as a top priority, and then the bots get secondary priority.”
The hosting platform also works at “every single layer of the stack,” allowing it to appropriately address each type of bot efficiently.
“We detect bots at the edge and then we can actually hand them content at the edge or block the bad bots right there at the edge,” McKee said. “Then there are others that we know we need to actually let all the way through. They need fresher content, not cache content.”
McKee said WP Engine partners with Cloudflare and leverages the cloud platform’s tools to detect bots and perform caching at the content delivery network “layer.”
“We are constantly looking at their leading edge solutions and working with them to improve their solutions at the same time as improving ours,” McKee said.
Crawling into the future. McKee said the ultimate impact the inundation of AI bot traffic will have on the internet is still unclear. She predicts that some companies and people may begin to create separate websites with specialized content for automated bots, similar to how organizations adapted to smartphone proliferation by introducing websites optimized for mobile devices.
“I would imagine we’ll see some of that in the beginning and then the systems will actually just catch up and the industry will kind of settle on a norm,” McKee said. “What that norm is, I don’t know yet.”