ABSTRACT: This blog, the second in a series, explores the mix of infrastructure types that support modern AI.

Read time: 5 mins.

Sponsored by Exasol

Forget the hype. Artificial intelligence thrives on premises as much as the cloud. In fact, organizations divide AI projects and workloads equally across their own data centers, public cloud platforms and hybrid environments. This trend reveals a paradox: even as AI adopters embrace disruptive innovation, they control as many variables as possible.

This blog, the second in a series, explores the mix of infrastructure types that support modern AI. It builds on the framework described in our first blog for analytics overall:

• On premises infrastructure offers control of cost and security as well as data sovereignty.
• Cloud infrastructure offers scalability, flexibility and access to advanced AI tools.
• Hybrid environments strike a balance, mixing and matching resources to optimize distinct projects or workloads.

This blog asserts that data and AI teams can achieve the best business outcomes by applying this framework to each component of the AI lifecycle, from data engineering to model development and production. The third and final blog in our series recommends best practices to optimize hybrid environments in particular.

Survey findings

Let’s start with survey findings from a new BARC report. My colleague Shawn Rogers and I asked 318 data, AI, IT and business leaders where they host eight components of their AI projects: data sources, data preparation, feature engineering, model training, model evaluation, model inference, retrieval-augmented generation (RAG) and production applications.

The high level numbers show an even split. On premises infrastructure garnered 34% of overall responses, compared with 33% for public cloud and 33% for hybrid. That’s a surprise given that AWS, Azure and Google are investing billions to become the one-stop shop for all things AI. Organizations clearly have a different view of their options than these cloud hyperscalers do.

Where Organizations Host Their AI Workloads (n=318)

Now let’s map our components to the three stages of the AI lifecycle: (1) data engineering, (2) model development and (3) model production.

We see subtle variations on the 34-33-33 split.

• Data and AI teams favor on-premises infrastructure for the first stage of the AI lifecycle, which includes data sources, data preparation and feature engineering.
• However, they favor hybrid and cloud infrastructure for the second stage of model development (i.e., training and evaluation); and the third stage of model production (production applications, model inference and retrieval-augmented generation (RAG).

Data and AI teams favor on-premises environments for data engineering and hybrid/cloud for model development and production

On premises

In part, the higher role of on-prem infrastructure for early lifecycle stages reflects the simple fact that many or even most raw model inputs remain on premises due to data gravity and migration complexity. This is true for semi and unstructured data such as documents, emails and images in particular. Many organizations have years of that historical data, rich with potential insights, sloshing around their own data centers. The simplest and most cost-effective option is to maintain those sources, prepare that data, and define those features on premises.

More broadly, the question of control affects the entire lifecycle. Organizations seek to reinvent their businesses and gain sustainable competitive advantage by applying AI models to their proprietary datasets. Given the high stakes, many adopters won’t incur additional risks with variable cloud expenses or volatile sovereignty requirements. They want to control each aspect of their AI projects, from data engineering to model development and production—along with the compute and storage underneath. Organizations in regulated industries such as financial services, healthcare and the public sector often have this risk-averse mindset.

Many AI adopters won’t incur additional risks by using the cloud. They want to control each aspect of their AI projects

Cloud

Still, the cloud holds undeniable benefits for others. Cloud hyperscalers help organizations access a rich ecosystem as they develop AI models and push them into production applications. This includes libraries of myriad starter models, from machine learning to predictive machine learning, that data and AI teams can download, train and evaluate based on their proprietary features. It also includes specialized tools for refining models, building AI applications and then monitoring and optimizing everything in production. In addition, hyperscalers help manage RAG workflows that retrieve relevant data, then augment user prompts to help GenAI models generate trustworthy outputs. Cloud platforms have expertise in all these areas that benefits AI adopters.

The broader advantages of cloud computing also apply here. Organizations gain economic flexibility by renting rather than buying infrastructure, then scaling resources up or down on demand. They also gain the flexibility to start, stop or change projects as needed. This helps data and AI teams experiment, iterate and learn without making significant long-term commitments. Such flexibility appeals to midsized or small organizations that are new to AI.

Cloud platforms offer a rich AI ecosystem as well as economic and resource flexibility

Hybrid

As the name suggests, hybrid environments include a mix of cloud and on premises resources. This mix enables AI adopters to strike a balance for distinct elements and workloads. Suppose a manufacturer has factory log files on premises but shipping records in a cloud lakehouse. The manufacturer decides to prepare data inputs, refine features and train models in both locations as part of a hybrid architecture. And the production environment depends on the model: preventive maintenance models for factory equipment operate on premises, but inventory optimization models operate on the cloud. The manufacturer rebalances as requirements change: when it upgrades to a video-based maintenance technique, it must migrate source data to the cloud to take advantage of cloud-based GPUs.

This manufacturer hosts its AI projects both on premises and in the cloud, depending on the use case

Promise and peril

Our interconnected world offers the promise of groundbreaking innovation—and the peril of missing out. To meet sky-high business expectations, AI adopters must consider the distinct benefits of different approaches—cloud, on premises and hybrid—for different projects and workloads. Data and AI leaders can use the tradeoffs described in this article to chart the right path for their organization. To learn more, also check out Exasol’s blogs about the advantages of on premises software and the motivations for on-prem platforms compared with the cloud.