A question we get asked regularly at The Duckbill Group is whether we can share benchmarks around what other organizations spend on AWS. The question takes several forms, such as:
- “What is a typical industry cost per transaction?”
- “How much should I be spending on $service?”
- “What’s the right amount of idle CPU across a fleet of systems?”
- “What do other companies spend on prod vs. non-prod?”
These are well-intentioned yet ultimately misguided questions.
The first issue is that external benchmarks tell you what the mediocre companies do, not what the best of them do. If your goal is to be a competitive leader, you’ll need to blaze your own trail.
We don’t provide benchmarks because of the second issue: They just aren’t that useful and don’t make sense without context. After all, your company has a unique product design and competitive business strategy, and your spend should reflect that. The idea of sharing a “typical industry cost per transaction” is farcical, because there’s no such thing as “typical” once you reach the level of individual businesses.
Benchmarks ignore product design
Let’s take, for example, a company whose product spends a huge amount of time crunching numbers of some sort. Assume it has done optimizations to really pack that compute density. Its typical idle CPU is going to be very low — which makes sense because it’s running a CPU-bound workload and it’s already made efforts to improve the density of its compute.
Taking that idle CPU benchmark to a company whose product is memory-bound doesn’t make any sense. Even choosing instance types that are heavier in memory capacity will still result in high memory utilization and low CPU.
Many workloads are not strictly CPU-bound; plenty are constrained by memory, IO, or network. But even taking into account all of these, a benchmark from some other company still doesn’t work for yours because the workloads aren’t the same. That company builds different products than you and in different ways. Even if you were direct competitors with a similar product, you made decisions on trade-offs that it made differently. This holds true even if we were to look for averages across multiple companies of similar products–-there’s just too many different choices made to result in anything meaningful for you.
The same goes for decisions around how much you spend on certain services — your product design dictates that, and you won’t find meaningful answers by looking outside.
Benchmarks ignore competitive decisions
While we’re on the topic of trade-offs and decision-making, let’s talk about prod vs. non-prod spend.
Non-prod spend encompasses dev environments, QA, and other environments that aren’t production. The distinction matters for finance purposes: Production gets categorized under cost of goods sold (COGS) while non-prod spend is under research and development (R&D). These are two very different buckets for the finance team.
Asking what another organization spends on R&D to make a decision about whether you’re spending the right amount is just as misguided as the resource constraint example above. These are competitive decisions! Do you really want your competitors dictating how much you should spend on R&D?
I’ve known many companies that go all-in on R&D spend for years at a time. I’ve known companies that spend years at the other end of the spectrum, simply trying to maintain what they’ve got without innovating further. Some companies go back and forth. I’m not one to judge whether any of them were right or wrong — that’s for their executives and the market at large to decide. Certainly, relying on external industry benchmarks isn’t a healthy approach.
Who cares whether your cost per transaction is a penny more than your direct competitor? Maybe you’ve got a better product!
Who cares that you’re spending 5% of revenue on R&D? Maybe your product is in such a mature position that increasing it wouldn’t move the needle. Maybe 5% is exactly the right number for your company.
These are inherently internal decisions guided by strategy. Don’t let your competitors dictate where and how you invest.
Compete against yourself, not others
We’ve helped a number of organizations develop KPIs to track how they’re doing. A great many of those KPIs are highly specific to the company, measuring the things that are unique to their particular environment and the things they care about.
Among our most top-performing clients, one of the things that stands out is that the best of them are competing against themselves, not against some externally set benchmark.
After all, by the time it becomes an external benchmark, all of your competitors were there already.
