“It’s ironic that we are the group that measures everything in the organization, but we don’t have a good way to measure our effectiveness.”

This is what Eric Colson told me when I asked him how he measured the success of his BI and analytics team at Netflix. (Eric is now chief data officer at Stitchfix.) When I pressed him, he said the best empirical measure he could find was the number of times business unit heads mentioned his team at their operational review meetings. If the business heads told executives that they were partnering with his team to deliver a project that would help them achieve their group’s strategic objectives, Eric took that as a good sign that they valued his team’s contributions. “We must be doing something right if they keep wanting to work with us,” Eric said.

For most BI managers, measuring success is an afterthought. It’s hard enough to get approval for a project and deliver results without having to kick off another project to measure your team’s performance. And if you do have the time and interest, what exactly do you measure?

Usage tracking. Most BI managers track usage to gauge performance and value. They monitor how many users have BI licenses, how often they log in, how many reports they run on average, how many queries they run against which data elements, and so on. But high usage doesn’t necessarily mean users are getting a lot of value or value commensurate with the investment in BI. Although you might have 1,000 users, only 25% log in weekly, and when they do, they only run one report, which they look at for 10 minutes. So, there is a lot of activity, but very little uptake.

Surveys. Some more ambitious BI managers send surveys to BI users to gauge their satisfaction with the BI tools and reports. Unfortunately, from what I’ve seen, the response rate to these surveys is pretty dismal (but that’s true for all surveys these days) which means results are potentially skewed. Typically, you only hear from those who are really happy or not. But the real value comes from the great unwashed masses who didn’t respond. So how good a measuring stick is a survey?

Social media analysis. One BI manager I know wants to add social media features to their BI reports, such as giving users the ability to rate, comment, and share on reports with their peers. Then using social media analytics, they can evaluate the value of each report, using both empirical and subjective data, and by extension, the value users are getting from BI deliverables.

Spreadmarts. In the past, I’ve jokingly said that a BI program’s success is inversely proportional to the number of spreadmarts in their environment. The theory is that the fewer renegade data shadow systems exist in the company, the more likely users are getting value from the BI team’s standard reports, dashboards and self-service environments. Of course, this means a BI manager has to find and monitor all the spreadmarts in their organization. But this is hard to do, like playing whack-a-mole. As soon as you discover one and consolidate it, three more spring up in other areas of the organization.

Cost efficiencies. The best BI managers track the costs of making decisions. Before they initiate a BI project, they establish a baseline that measures the cost of hardware and software licenses and the time analysts spend accessing data rather than analyzing it. After completing their project, the BI managers remeasure these items to gauge the financial lift of the BI project.

 Companies that implement BI for the first time can usually wring lots of costs from their decision making processes by making things more efficient. But companies with mature BI programs don’t have this luxury. BI managers here must justify continued investment in the BI program by its strategic value. This usually involves measuring the value of better decisions, of mission-critical processes, or more informed workers. This is not easy, but can be done. Unfortunately, the results can always be disputed by a cynical-minded executive.

Full circle. And this brings us back to Eric Colson. Perhaps tracking the number of mentions the BI team gets in an executive meeting is not an exact science. And perhaps it’s a bit unseemly or egomaniacal to track such mentions. But as far as I can tell, it’s the best metric we have for truly measuring the value of a BI program.

Let me know what techniques you’ve used to measure BI success. (And you can learn more about Eric Colson in my book, “Secrets of Analytical Leaders: Insights from Information Insiders” available at Amazon.com.)