The World of BI Turned Upside Down

Until recently, I knew how to design a business intelligence (BI) system. Today, I’m not so sure. Everything is in flux. All the old rules don’t seem to apply. People are using new approaches, techniques, and tools to build data analytics systems. And much of what they are doing is compelling.

 I attended my umpteenth TDWI conference last week. You can always count on TDWI to deliver BI orthodoxy. But even there, my BI compass began to quiver. It began to look and sound more like a Strata/Hadoop event (see Ten "Design First" Principles from Strata/Hadoop.”) than a traditional BI/DW conference. The overarching mantra was: “Out with the old, in with the new.”

Here are some of the more interesting things I heard at TDWI, a mix of tips, tricks, and provocative new thinking:

  •   “There is no need for a dimensional model.” (TRANSLATION: the BI tools are so good they can compensate for inferior data designs.)
  • “There is no need for ETL tools.” (TRANSLATION: use Spark for ETL with cloud or Hadoop environments.)  
  • “You don’t need a relational database.” (TRANSLATION: You can do all your data science work in S3 and Spark.)
  • “Code is not the enemy.” (TRANSLATION: write ETL code in Spark and manage it in Git Hub; it’s “liberating”.)
  •  “We don’t move data.” (TRANSLATION: we build logical views inside Hadoop to support analytic use cases.)
  •  “We design from physical to logical, not logical to physical.” (TRANSLATION: we we ingest raw data, then build logical views for each use case; we don’t model first.)
  • “We design for secondary use cases, not the primary one, which has a limited shelf life.” (TRANSLATION: we ingest and store atomic-level data so we can re-purpose it as new needs arise.)
  • “Your data architecture is as important or more than your data model.” (TRANSLATION: how data sits in your file systems is important or they won’t find, make sense of it, or navigate it.)
  • “Architecture is way more important when you move to the cloud.” (TRANSLATION: if you don’t configure your cloud environment correctly, you could pay a lot more than you need to.)
  •  “The key is flow: the business flow, the architecture flow; business likes attaching to flows.” (TRANSLATION: we’ve automated our data pipelines from ingest to delivery using metadata tagging and synchronization.)
  •  “A data catalog is the nervous system of an analytic ecosystem.” (TRANSLATION: it dynamically detects, tags, and synchronizes metadata across an ecosystem.
  • “We ingest a new data source in a week.” (TRANSLATION: we automatically detect schema changes in the source and update our target views and metadata.)
  • “We use DevOps to automate our development and production processes.” (TRANSLATION: we use version control and change management techniques to rapidly move through development, test, and production processes.)
  • “Each of our 13 scrum teams functions like a spanner.” (TRANSLATION: each scrum team handles the entire data analytics pipeline, from ingestion to visualization, in a narrow business domain.)
  •  “Every quarter, our 15-20 business teams collectively prioritize their needs based on our capacity.” (TRANSLATION: Business teams spend two days every quarter presenting their goals, mapping them to IT capacity, and prioritizing their requirements, if required.)
  •  “We want more demand than our capacity to handle it.” (TRANSLATION: excess demand requires us to be more efficient and forces the business to self-prioritize their requests.)
  • “Every time the business reorganizes, we use that as an opportunity to dismantle our legacy systems and build new.”
  • “Business users are intimidated by Hadoop.” (TRANSLATION: we integrate a legacy environment with Hadoop to give power users an on-ramp to our new architecture.)
  •  “Use graph database for MDM.” (TRANSLATION: it’s a more natural way to manage and synchronizes references.)
  • “Applications are dependent on analytics.” (TRANSLATION: we need DevOps to coordinate application and analytics development.)
  • “The same person who engineers the data analyzes and deploys it. (TRANSLATION: data engineering, data science, analytics are converging.)
  • “Either you evolve and change, or die.” (TRANSLATION: embrace Hadoop, Spark, and the cloud.)

The times are changing. Every organization that has implemented a BI or data warehouse more than five years ago needs to undertake a modernization project. Otherwise, it risks becoming antiquated, costly, and irrelevant. The only constant is change, and now is the time to change! 

Wayne Eckerson

Wayne Eckerson is an internationally recognized thought leader in the business intelligence and analytics field. He is a sought-after consultant and noted speaker who thinks critically, writes clearly and presents...

More About Wayne Eckerson