ABSTRACT: Explore our four primary criteria for evaluating conversational BI products

Sponsored by ThoughtSpot

Conversational business intelligence (BI) will soon dominate the game of analytics. And analytics teams need to find the new goalposts.

Conversational BI uses generative artificial intelligence (GenAI) to guide and automate various BI tasks through chatbots. Most major BI tools such as ThoughtSpot now include conversational BI chatbots that complement and even diminish familiar graphical interfaces. They use language models to interpret, respond to, and act on natural language prompts from humans. Such capabilities increase productivity for analytics power users and provide self service to casual users. 

Achieving those benefits, of course, depends on selecting the right product. This blog explores four evaluation criteria for conversational BI: breadth of functionality, ease of use, human governance, and ecosystem support. We start by defining the key capabilities and who uses them. To keep things simple, we refer to BI products that contain conversational BI as “conversational BI products.”

Defining the market

GenAI is a new technology that generates digital content such as text, images, or audio, often with stunning speed and sophistication, after being trained on a corpus of existing content. The most popular form of GenAI centers on a language model (LM), a type of neural network that interprets, summarizes, and generates text. BI vendors are building LMs into their products to help users perform a range of tasks:

• Data preparation. Prepare data for analytics by filtering, enriching, cleaning, formatting, and structuring it.

• Cataloging. Organize metadata such as file names, database schemas, and table attributes to assist in data discovery, assessment, and governance. 

• Analytics. Analyze data and visualize the outputs in reports and dashboards or feed them into applications, workflows, and alerts.

Both power users and casual users leverage conversational BI to simplify and accelerate how they interact with data. Their primary use cases include the following.

• Power users—including data analysts, data scientists, data engineers, data stewards, and application developers—gain productivity. They use conversational BI to develop, debug, and document code, evaluate analytical outputs, and create predictive features. 

• Casual users, including data consumers and data explorers, gain self service. They use conversational BI to generate and explain reports, models, and insights for them—then trigger application tasks to take action.

Evaluation criteria

Now we explore our four primary criteria for evaluating conversational BI products: breadth of functionality, ease of use, human governance, and ecosystem support. Here is how analytics leaders should probe vendors in each criteria. 

1. Breadth of functionality. Ask your BI vendor to demonstrate how they help both power users and casual users accomplish their highest-priority use cases. Think creatively about potential future use cases that your analytics community might need to support over the next 18 months based on business requirements. These might also require capabilities that complement GenAI, such as advanced search, coding assistance, or embedded analytics. And of course, you’ll need traditional BI capabilities such as interactive data visualization and support for diverse data sources.

2. Ease of use. Power users and casual users alike need simplicity wherever possible as they converse with data. You can evaluate ease of use on several dimensions. Does the conversational BI chatbot understand your team’s natural language? Can it learn technical slang or jargon? How much training is required to become proficient with the chatbot? Look for auto-completion of prompts, recommended questions for users to ask, and explanation of KPI alerts. Users also should be able to pose questions in an iterative fashion as they perform exploratory data analysis. And they should receive natural language descriptions that business decision makers can easily understand.

3. Human governance. Language models can generate false statements, leak sensitive data, and embarrass companies. Analytics teams need their conversational BI product to mitigate these risks based in no small part on human feedback. Evaluate how your product invites and responds to user indications about the accuracy, relevance, and confidentiality of its outputs. Users can provide these indications by clicking thumbs up or down, editing content, or flagging it as non-compliant. Conversational BI should calibrate future responses based on this feedback, ideally by fine-tuning the underlying language model. This can increase response quality over time, for example by learning how common business terms relate to companies’ domain-specific datasets. 

4. Ecosystem support. Like all software, conversational BI lives and breathes in a vibrant ecosystem. Evaluate how these conversational BI capabilities support the critical elements of this ecosystem. Do they support both SQL and Python? Can users replace ChatGPT from OpenAI with alternatives such as Gemini from Google? Also probe vendors about their ability to have chatbots share outputs with third-party dashboards, AI/ML notebooks such as Jupyter, or developer platforms such as GitHub. While these might be stretch goals for your initial deployment, over time your power users in particular will want the ability to build and orchestrate multi-faceted workflows across elements such as these. 

By evaluating products’ breadth of functionality, ease of use, human governance, and ecosystem support, your analytics team can compete effectively in the dynamic game of conversational BI. Join Eckerson Group’s upcoming webinar with ThoughtSpot and Google Cloud on March 27 to learn more.