ABSTRACT: As enterprises grow more dependent on the cloud and as the economy convulses, FinOps will soon become mandatory.

Sponsored by Yellowbrick

The cloud gives enterprises a rich, flexible platform on which they can innovate and grow. But it has risks. Running your business on elastic infrastructure, especially compute clusters, can lead to budget-breaking bills at the end of the month. Risks like these underscore the need to govern cloud-related costs, especially in an uncertain economy.

FinOps might be able to help. This emerging discipline enables IT and business teams to collaboratively control the cost and increase the value of cloud-based projects. FinOps instills best practices, automates processes, and assigns accountability to stakeholders that range from business and finance managers to IT and data engineers. This blog explains how a FinOps program helps govern the cost of analytics projects.

FinOps for Analytics Projects

FinOps guides cross-functional teams as they forecast, measure, and account for the staff time and IT resources that go into an analytics project. It helps them oversee the project lifecycle, including design, operation, and adjustments. And it helps them control the cost of cloud-based applications as well as storage, compute, and network resources. Let’s walk through the details, working from the bottom up in the diagram below.

Cloud Usage. Cloud-based analytics projects depend on applications, storage, compute, and network resources. In most cases, their costs vary based on usage, although some vendors, such as Yellowbrick, offer the alternative of fixed-resource subscriptions for its cloud-based data warehouse.

• Applications. Data analysts and data scientists rely on a wide range of applications. They might use a data warehouse from Yellowbrick to build BI reports, a machine learning platform from Databricks to build ML models, or various operational applications to embed analytical functions in production workflows.
• Storage. Data teams pour all kinds of multi-structured data into cloud object stores such as Amazon S3 and Google Cloud Storage. While storage is not the biggest cloud cost, the tendency of teams to store all raw data and multiple copies of some datasets can strain enterprise budgets.
• Compute. Elastic compute services such as Amazon EC2 and Azure Virtual Machine can cost a lot of money when queries or other processing jobs spike. Data teams need to think hard about the cost implications of supporting variable workloads.
• Network. Data teams rely on network services such as Amazon VPC to isolate resources, Amazon API Gateway to manage data access, or Amazon Elastic Load Balancing to distribute traffic. Analytics projects that straddle multiple data stores or locations can cause spikes in these charges as well.

Project Lifecycle. Data teams consume all these resources as they design, operate, and adjust the data pipelines, applications, and environments that support their projects.

• Design. Data engineers select and configure the data stores, compute clusters, algorithms, and other elements that comprise data pipelines. Data scientists and developers, meanwhile, design and build applications that consume data.
• Operate. Data, ML, or CloudOps engineers test these elements, then roll them into production. Along the way, they consume compute cycles, storage capacity, and so on.
• Adjust. These engineers reconfigure pipelines, allot new resources, and make other changes to optimize results.

Project Oversight. Now comes the heart of the FinOps program: project oversight that helps stakeholders forecast, measure, and account for their time and resources. They use various commercial visualization and observability tools, as well as cost calculators and optimizers to perform the tasks described below.

• Forecast. Once the business manager and analytics lead define their use cases, they set their budget with the finance manager. Then they work with the data, ML, and/or CloudOps engineer to forecast the required time and resources needed, and tweak project plans as needed to stay within budget.
• Measure. Engineers monitor cloud usage as they operate their pipelines, applications, and other elements in both the test/dev and production stages. They track the cost of application usage, compute clusters, and network resources in particular, casting a wide net to ensure they measure all the data marts, databases, etc. for each project. They also monitor workload behavior to predict and trigger alerts for cost overruns. When teams spot issues, they isolate the cause—perhaps an untrained user, misbehaving application, or configuration error—and make the necessary project changes to minimize future risk.
• Account. Engineers report cloud operating costs back to business and finance managers, who hold team leaders accountable for those costs. This accountability strengthens budget planning, operations, and cost management.

FinOps teams also must master the pricing models of various cloud services. Business owners need to understand those models’ implications for project ROI, and IT stakeholders need to understand their implications for resource optimization. Cloud pricing models can influence how they adjust resources in the near term, and how they adjust architectural approaches in the medium and long-term.

As with all worthwhile endeavors, adopting FinOps can feel like swimming upstream. It’s hard to measure this stuff consistently, and it’s just as hard to change behavior. But as enterprises grow more dependent on the cloud, and as the economy convulses, this endeavor will soon become mandatory.

To learn more, check out Eckerson Group’s webinar with Yellowbrick Data last month.