Cost assessments are sort of our bread-and-butter here at Duckbill, both in identifying ways to reduce costs and, more importantly, assessing an organization’s ability to manage its own costs. About two years ago, we decided to formalize our assessment methodology into a structure we call Cloud Cost Compass.
Since then, we’ve run this assessment on many dozens of organizations, with monthly AWS spend ranging from $100,000 per month up to several tens-of-millions per month. Today, we’re publishing our assessment methodology.
Two interesting takeaways
We learn two interesting things from our experience running this assessment on so many organizations.
First, high-performing organizations are not high-performing everywhere, and vice versa. The reality is that even in low-performing organizations, there are always pockets of high performance. In high-performing organizations, there are always pockets of weakness.
Second, organizations have a tendency to regress their performance during periods of business growth, creating a need to rethink their approach to managing costs. This makes sense, of course: innovation is inherently wasteful, and growth is often better than cutting expenses. As growth slows down, the need for efficiency comes back into play.
Criteria
Cloud Cost Compass contains six areas to assess.
| Compute Optimization | Compute Optimization is a measure of compute density, intentionality of choices concerning various kinds of available compute resources, and use of ephemeral compute resources. |
| Data Optimization | Data Optimization measures how well an organization understands and leverages the available types of AWS data storage and understands and optimizes for the architectural and financial impacts of data transfer. |
| Cloudiness | Cloudiness is a measure of how an organization uses AWS, employs modern cloud practices, and level of continual improvement. Does the organization treat AWS simply as another data center by only leveraging the most basic primitives of AWS? Or does the organization make extensive use of the higher-order services? The more the latter is true, the more optimized cloud costs tend to be. |
| Resource Purchasing Commitments | Resource Purchasing Commitments measures the extent to which an organization takes advantage of the significant discounts AWS provides for customers who are willing to commit to a base standard of usage. This includes Reserved Instances, Savings Plans, and private pricing agreements (EDP/PPA). |
| Cost Visibility | Making costs more visible across the organization is an important aspect in understanding and controlling those costs. Companies that make investments in visibility strategies fare better with understanding and managing AWS costs. This criteria also covers cost allocation. |
| Collaborative Cost Management | Measures the strength of the collaborative relationship between Finance and Engineering when it comes to AWS cloud costs, particularly decisions and understanding related to unit economics, forecasting, business seasonality, and each department’s ability to influence spend. |
Rubric
We score an organization on a 1-5 scale for each area. We intentionally leave 2 and 4 empty to provide room for in-between scores. The rubric is subjective, as it’s designed to be used and interpreted by people with expertise in the subject matter rather than run by some automated system or a junior-level team member.
| 1 | 2 | 3 | 4 | 5 | |
| Compute Optimization | Nearly all of our infrastructure is on long-lived EC2 instances. We treat AWS as just another datacenter. | Extensive use of AWS-managed services relevant to the environment’s use cases. Extensive use of IaC tools and practices. Extensive knowledge across the Engineering org with respect to AWS/cloud design and architecture principles and practices. Established architectural review processes. | Instance types are intentionally chosen Anything that can have a lights-out has one Serverless is used as much as is practicable High utilization of low-cost process architectures (eg Graviton) Maximized compute density | ||
| Data Optimization | We almost exclusively use EBS storage or EC2 instance-backed storage. We leverage S3 only for backup purposes. No S3-IT, no EBS/S3 lifecycles. Basically, AWS defaults. No understanding of data transfer or data lifecycle. | Consideration given toward use of various S3 storage classes Efforts made toward understanding and classifying data usage patterns Efforts made toward understanding data transfer patterns | Extensive/predominant use of lowest-cost S3 storage classes. Demonstrated understanding of data usage patterns (instance store, EBS, S3, etc). Data transfer costs are well-understood and architectural decisions are data-driven. | ||
| Cloudiness | We use only the most basic AWS primitives, like EC2, S3, EBS, and RDS. Use of old-school orchestration systems (eg, OpenShift) or hypervisors (eg, VMware) | Moderate use of AWS-managed services relevant to the environment’s use cases. Moderate levels of IaC tooling and practices Moderate levels of knowledge across the Engineering org with respect to AWS/cloud design and architecture | Extensive use of AWS managed services relevant to the environment’s use cases. Extensive use of IaC tools and practices. Extensive knowledge across the Engineering org with respect to AWS/cloud design and architecture principles and practices. Established architectural review processes. | ||
| Resource Purchasing Commitments | We have not purchased a Reserved Instance or Savings Plan yet. Everything runs on-demand. No private pricing contracts, though eligible for them. | Ad hoc purchasing of RIs/SPs for eligible services. No assigned owner for contracts, but contracts are in place. No assigned owner for RIs/SPs. RI/SP coverage consistently >75% and utilization consistently >80% | RI/SP coverage consistently >90% and utilization consistently >90% We have an assigned owner whose job is to forecast and maintain RI/SP purchases. We have an assigned owner for the AWS bill and contract and have high confidence we have the best-for-us deal possible. | ||
| Cost Visibility | I get a bill every month and I pay it (sometimes on-time!) | One sole owner of AWS spend, but it’s at a VP/CTO level. No defined process for spend visibility/management. Tagging strategy in place but inconsistently applied with low to moderate coverage. Low-to-moderate ability to map spend to teams/products/services. Teams do not contribute to managing their spend in an active way. | We have a dedicated role/team for understanding spend. We can accurately map spend to teams/products/services/projects with ease. These teams/products/projects are aware of their spend and take an active role in managing it. Strong understanding of our cost model and how that influences our AWS private pricing and contractual commitments. | ||
| Collaborative Cost Management | Relationship between engineering and finance is either non-existent or actively hostile. Finance has no data on cloud spend beyond the AWS bill. Finance has no influence on cloud spend. Engineering has no influence on budget. Unit economics are undefined. There is no cloud spend forecast. No understanding of business seasonality as it applies to cloud cost. | Engineering and finance regularly communicate about cloud spend, primarily to share information. Finance periodically gets data on cloud spend from engineering. There is an established, regular way to set goals on cloud spend. It’s not collaborative, but it’s functional. You have a unit economics story that somebody set, but it’s not clearly understood how you got to it or how accurate it is. It’s not really used outside finance. You have an unreliable forecast. Basic understanding of business seasonality as applied to Engineering and Finance concerns. | Engineering and finance effectively communicate about cloud spend. Finance has a self-service way of looking at cloud spend by cost center, customer, etc. Finance and engineering actively collaborate to influence/set goals on cloud spend in response to broader organizational pressures. You have a unit economics story, both eng and finance know how you got it, and either eng or finance can influence it. Unit costs play an active role in decision-making across the organization. Engineering and finance actively collaborate to continually refine their cloud spend forecast. Demonstrated ability to tie business seasonality with Engineering and Finance concerns. |
Want us to assess your organization?
Just reach out and we’ll work to get you scheduled.
