Fanatics

Fanatics streamlined cloud utilization, saving millions annually

The sports merchandise e-commerce company engaged Duckbill Group to perform a major analysis of its business-critical data processing and analytics environment, uncovering millions in savings without impacting business operations.

Overview

• Client: Fanatics, a global sports merchandise retailer
• Number of employees: Over 7,500 globally

Situation

This fast-growing e-commerce retailer needed to improve the efficiency of its large-scale data processing platform, improving its cost basis and increasing profit margins.

Solution

Duckbill Group recommended specific provisioning and utilization changes that saved Fanatics millions of dollars annually and delivered a new framework to easily monitor and control costs going forward. 

About Fanatics

Fanatics, the global leader for licensed sports merchandise, is a new breed of retailer – part tech company, part logistics expert, part e-commerce company, and part manufacturer – that combines exclusive deals with leagues and teams with on-demand production. Its agile supply chain increases the speed-to-market of high-quality fan merchandise distributed globally across all retail channels.

One of the strategies that make Fanatics so successful is its ability to predict and respond to demand for sports merchandise for a specific team or player. For example, if an unknown rookie player makes an incredible play and gains popularity among fans, Fanatics can track in real-time the new demand for that player’s products and instantly offer a jersey sporting the newcomer’s name. By tracking signals via social media, sports news, and searches on its network of hundreds of online stores, when fans come shopping Fanatics is ready to deliver. 

A renewed focus on cloud operational costs

For Fanatics, driving new revenue is a top priority. As the company planned for the year ahead, even with strong growth forecasted, the technology organization knew that improving efficiency in the cloud was key to improving margins. 

Like most e-commerce companies, the engineering teams are primarily focused on shipping new features and delivering a great customer experience, and the platform engineering team’s primary job is to support them in achieving a common business goal. 

For Johnny Sheeley, Fanatics Director of Cloud Engineering, improving the efficiency of the cloud infrastructure was a top priority.

The first point of investigation: The utilization of the large number of data-processing instances that powered the smart downstream systems. Daily, thousands of ephemeral instances were spun up for batch-processing data from many different partners, as well as from Fanatics’ own internal services. This added large quantities to the data lake. 

Using a cloud cost management tool, Fanatics monitors usage stats, but the tool doesn’t offer guidance on how to manage individual resources to high-level business KPIs. Moreover, the engineering team didn’t want to veer away from product development priorities. 

“We aren’t going to cut the velocity of our new feature delivery and that’s where our teams are focused,” said Sheeley. “So we decided to bring in the cloud experts at Duckbill Group who could give us a fresh perspective on our cloud architecture and help us streamline our cloud infrastructure costs.” 

Custom analysis identifies cost savings in automated processes

Duckbill Group analyzed the recorded usage data on hundreds of thousands of data-processing instances. The team designed a strategy that would dramatically reduce Fanatics’ AWS spend: reduce the number of machines used in overnight batch processes, as well as streamline each machine’s resource allocation based on expected workload. To get there, the specifics for this action plan started with sophisticated data analysis and a deep understanding of Fanatics’ unique data usage model.

While the cloud cost management tool could quantify the number of cloud instances, it took the special data analytics skills and custom analysis from Duckbill Group’s engineers to design a solution. The standard procedure for this type of large-scale batch data processing is to create ephemeral instances, run the jobs, and then terminate the instances, all automatically. With thousands of short-lived instances to analyze there isn’t a “dashboard view” of utilization that points to a workable solution.

“When Duckbill Group’s report said we’d save millions of dollars annually everyone could see this would be a big win.”

John Sheeley, Fanatics Director of Cloud Engineering

In order to go deeper, Duckbill Group designed a custom process to analyze these data-intensive tasks. The team looked at the utilization of the compute clusters for batch jobs and set goals that the Fanatics teams could track to better maintain efficiency going forward. These recommendations were tiered by effort and cost-savings so the Fanatics team could prioritize the improvements and tackle them over time. 

The first round of changes was easy to prioritize. “When Duckbill Group’s report said we’d save millions of dollars annually everyone could see this would be a big win,” Sheeley said. “With the clarity of the recommendations and the obvious cost savings, the application teams were happy to negotiate priorities. And, it turns out, the changes were relatively easy to make and didn’t take very long to implement.” 

Duckbill Group’s report also identified new key performance indicators (KPIs) for this core service, determined their current performance level, and then set new targets for performance based on similar companies. These new benchmarks will help the Fanatics team measure progress over time and maintain cost savings. 

Bringing predictability to cloud spend

“Before the cloud, you had to justify your investments in hardware and software purchases and explain why the capacity was necessary,” Sheeley remembers. “The whole reason people adopt AWS is to avoid that, right? But it’s way too easy to get hooked on AWS and all of a sudden, your spend is out of control. 

“One of the greatest benefits we got from our engagement with Duckbill is we now have lifecycle management for every massive data bucket. And we now have a predictable spend.”

Sheeley isn’t finished working with Duckbill Group yet. Next, the teams will tackle architecture and performance improvements. “This first round was focused primarily on cost savings and some aspects of performance, and next we want to tackle more architectural decisions and upstream issues.”

Sheeley found Duckbill Group easy to work with, and especially appreciated the straightforward approach to reporting recommendations.

“They explained the approach that they would take, the things that they needed to be successful, and from there, they interviewed our internal teams to gather context and data from our environments,” Sheeley said. “I know the analysis they did was complex but what was great is the report was not. This helped a lot because I could say ‘Read this and we can save two million dollars.’ People could easily understand the impact.”