It was Nov. 15, 2022, and Swifties everywhere were poised in front of laptops waiting to buy tickets for her Eras Tour on Ticketmaster. The ticket seller had previously promised Swift’s team that it could handle the capacity, but its technology failed to keep up with demand, and many fans were left disappointed.
The company issued an apology, claiming that it is “working to shore up our tech for the new bar that has been set by demand for the…Eras Tour.” and said the traffic was quadruple what they expected, reaching 3.5 billion total system requests at its peak. Joe Berchtold, president and CFO of Ticketmaster’s parent company Live Nation, said in front of a Senate Judiciary Committee that as well as fans, the unprecedented influx came from bots and “industrial-scale ticket scalping.”
Technologists have been trying to solve the Taylor Swift problem long before last Fall. StackPath CEO Kip Turco told Protocol in 2021 that edge computing, where data is processed faster and more efficiently on servers physically closer to users, was the answer. Other methods of stemming the tide of scalpers and ticket timeouts include Ticketmaster’s Verified Fans program, but 3.5 million people preregistered, the largest in the company’s history.
Spencer Kimball, former Google engineer and current co-founder and CEO of Cockroach Labs, doesn’t work with Ticketmaster, but he understands how tricky it is to deal with that kind of increase in demand. It’s typically pretty hard to accommodate, Kimball explained to IT Brew. To always be able to react to four times your peak “feels like a waste of money,” Kimball said.
According to Kimball, updating legacy architecture for a business that you hope and expect to grow is complicated. The database is central to the entire operation. “That’s what keeps track of all of the seats and the tickets and the code so that multiple people aren’t coming in there and getting the same seat or someone’s using the same code 10 times and getting 10 tickets,” said Kimball.
Top insights for IT pros
From cybersecurity and big data to cloud computing, IT Brew covers the latest trends shaping business tech in our 4x weekly newsletter, virtual events with industry experts, and digital guides.
On top of the database is the application logic and all the services used to verify users and thwart nefarious bots. “Some of those things are easy to scale up,” Kimball said. “Especially in the cloud where I assume Ticketmaster is running.” It’s the database that becomes harder to scale. “You can’t have two different databases that don’t know what the other one’s doing when they’re all trying to balance the same set of seats,” said Kimball. That’s a recipe for disaster. And that, Kimball said, is “fundamentally what happened to Ticketmaster.”
Kimball argued that serverless architecture could be the answer to scaling up fast. “Even on the night of, you’re not having to add nodes into a system and trying to rescale it. It’s already built to accommodate this massive elastic scale.” Kimball explained that the technology in the Cockroach Lab database allows you to scale up a day or even a few hours in advance and then scale it down afterward.
While it makes sense for Kimball to recommend his own service, he argued that the best thing a company can do is to “plan for an architecture that can handle (ideally in real-time) an unprecedented surge without breaking. But all of that is just to weather the storm. And then you have to go and say, how do I redesign this UX so that these bad actors can’t get in?”