To improve your command of an e-commerce business, factory or telecom network, a common principle applies: build external data and operational triggers into the architecture you have today.

This is the essence of how to get started with Continuous Intelligence (CI). As defined in my last blog, CI refers to a set of functions – leveraging homegrown code or vendor products – that together automate real-time and historical analytics in order to directly adjust operations. It is not full CI until you assemble all the pieces. You can extend many types of architectures to achieve CI. Two primary types are architectures focused on business application data, and those focused on machine-generated data.

What Continuous Intelligence Does

Let’s start with the basics. CI functionality automatically ingests, transforms and analyzes data from many familiar sources. The analytics engine, often a stream processor, then recommends or automatically executes operational actions based on continuous assessment of all those signals. Figure 1 illustrates this functionality.

Figure 1. Continuous Intelligence Functionality

Architected well, CI functionality supports many use cases. For example, shipping companies might use CI to flexibly absorb supply chain shocks by continuously studying customer orders and RFID tags, plus external data such as virus alerts, regulatory changes or weather patterns. Retailers might use it to fine-tune prices based on purchase behavior or clickstream data, plus external data such as social media trends and regional telework habits.

Approach #1: Adapt Your Application Data Architecture for CI

The first common approach to CI extends existing architectures that focus on application-generated business data such as transactions, customer records, etc. Many of these architectures already use streaming to offer some, but not all, CI functions.

For example, databases, social media, Software as a Service (SaaS) or other application sources will publish business data via APIs, protocols or change data capture methods to streaming brokers such as Apache Kafka. The brokers sort that data and send it to stream processors. Stream processors use real-time rules or machine learning modules to filter and evaluate the data, and correlate it with historical data to understand patterns or outliers.

You typically need to add two critical functions – leveraging homegrown code, new vendor products or new features – to complete the CI picture.

First, you need to add some basic exogenous data – related to market trends, commuting traffic patterns, population health advisories, you name it – to put the analytics output into a broader context.

Second, you need to automatically recommend or generate real-time actions. To build on our retail example, a machine learning model might trigger a popup window to upsell their e-commerce customer on a related offer as they proceed to checkout.

Once all the CI functionality is in place, these application-data architectures can enable data scientists, BI analysts and business managers all to run the business faster and smarter. The latest streaming pipeline and analytics products from vendors such as Tibco help make this happen.

Figure 2 illustrates CI functionality in an application data architecture.

Figure 2. Continuous Intelligence in an Application Data Architecture

Approach #2: Adapt Your Machine Data Architecture for CI

The second major approach layers CI functionality onto architectures that focus on machine data.

Loosely defined, machine data is generated by computers rather than individuals. IoT equipment sensors, cloud infrastructure, security firewalls and websites all throw off a blizzard of machine data that measures machine status, performance and usage. In many cases the same math can analyze machine data for distinct domains, identifying patterns, outliers, etc.

Enterprises have well-established processes such as security information and event management (SIEM), and IT operations (ITOps), that process machine data. Security administrators, IT managers and other functional specialists use mature SIEM and ITOps processes on a daily basis.

Generally, these architectures perform similar functions as in the first approach, although streaming is a more recent addition. Another difference is that many machine-data architectures have more mature search and index capabilities, as well as tighter integration with business tasks and workflow.

Data teams typically need to add the same two functions to complete the CI picture. First, they need to integrate doses of contextual data to achieve similar advantages as those outlined above. Second, they need to trigger business processes, which in this case might mean hooking into robotic process automation tools. For example, a commercial bank could automatically freeze accounts accessed by perpetrators of suspicious activity that an SIEM tool identifies. Products from vendors such as Splunk help make this happen.

Figure 3 illustrates CI functionality in a machine data architecture.

Figure 3. Continuous Intelligence in a Machine Data Architecture

All the Pieces of the Puzzle

You can view Continuous Intelligence as a puzzle: the picture is not complete until you fit all the pieces into place. The good news is that most enterprises already have many pieces assembled, and vendors are making the additional pieces readily available.

My next blog will examine trade-offs and guiding principles to ensure your CI functionality hangs together.