ABSTRACT: Hybrid development teams are critical to the success of a data & analytics program. Data leaders need to nurture these team with knowledge flows.

This is the second of a multi-part series on data & analytics operating models. 

In the last article, I presented a high-level operating framework for aligning data and analytics resources in an enterprise. I argued that an effective operating model eliminates data bottlenecks, fosters self-service without creating data silos, and creates a data-driven culture. 

The operating model I presented contains three major components: enterprise data teams, domain-based development teams, and business domains, or blue, purple, and red groups respectively. (See figure 1.) The most important of these to the success of a data & analytics program is the “purple team.” Unfortunately, this is the one that data leaders most often ignore or shortchange. 

Figure 1. Operating Model Flows

The goal of purple (domain-based development) teams is to balance the enterprise’s need for standards, consistency, and economies of scale with the business unit’s need for agility, flexibility, and customization. Maintaining these polarities is challenging and costly. As a result, many companies skip the hybrid approach and focus on either a centralized (blue) or decentralized (red) approach. Eventually, they experience the downsides of either approach and flip to the other extreme, sometimes multiple times, without ever giving a hybrid approach a reasonable try. 

Data leaders who embrace a hybrid or “purple” approach to data development get the best results. Why? A hybrid environment immerses developers in a business domain where they learn its people, processes, and data, and consequently, build solutions faster and better. At the same time, they learn enterprise methods, standards, and tools that enable them to work smarter and more efficiently and deliver results that conform with enterprise data standards. 

In other words, a hybrid approach enables developers to work locally where they can have the most impact but develop uniformly to ensure data consistency and enterprise alignment. In this way, purple teams combine the best of centralized, IT-led approaches and decentralized, business-led approaches without the downsides of either.

A hybrid approach enables developers to work locally where they can have the most impact but develop uniformly to ensure data consistency and enterprise alignment.

Knowledge Flows

So, how do you build a hybrid development team that delivers the best of both worlds without any downsides? How do you build a purple team that delivers effective data solutions quickly and affordably without creating data silos, data bottlenecks, or conflicting data sets? There are many options, which we’ll examine in detail in our next article. But the major ones are: 

• Tiger teams that are permanently assigned to individual business units and backed up by a shared services center.

• Analytics centers of excellence in which data analysts and scientists are assigned to individual business units even though they might reside centrally. 

• Data domain teams (a la data mesh) that adhere to enterprise standards and governance policies and use enterprise data tools and platforms. 

• Embedded data analysts who are hired, evaluated, and managed by a director or manager of analytics at the enterprise level.

The key to making these models work is managing the knowledge flows between blue, purple, and red teams. Data leaders need to manage these flows to ensure the success of a hybrid or federated model. 

Top-Down Flows: Blue to Purple

Other than sourcing data, one of the most important roles of an enterprise data team—and one that few data leaders allocate staff time to support—is to transfer knowledge, standards, and best practices to the hybrid development team. Unless the enterprise data team shares knowledge and communicates standards, the purple teams will work sub-optimally and eventually create fragmented data that frustrate business executives to no end. 

Oversight. To avoid this fate, the heads of enterprise data team departments—namely, data engineering, data architecture, and data analytics—need to oversee hybrid teams. This usually entails 1:1 meetings with individual developers, daily and weekly standup meetings with individual teams, and quarterly retreats with all teams where they can establish and review goals, learn new techniques, and share experiences and knowledge. 

The enterprise data team also needs to manage data analysts and scientists, whether they are centralized in a center of excellence or embedded in business units. For example, a large urban hospital has two full-time managers who help hire, evaluate, coach, and manage the organization’s 35+ data analysts, who work for individual departments.

Train, coach, and support. The enterprise data team also needs to assign experts from each technical department to disseminate knowledge and best practices to members of purple teams. The experts should offer both formal training on the development methodology and tools and informal coaching and support through office hours, open labs, webinars, in-house events, and lunch-and-learn sessions.

Establish standards. It’s critical that the technical department heads establish standards and document them. For example, it’s important to establish a standard way to organize development teams, engage with business users, and develop and deploy code. Many companies choose Scrum or Kanban and borrow heavily from dev/ops practices. Other standards might define how to submit project requests, how to manage business logic, how to implement data quality rules, and how to propose new policies or procedures.

The enterprise data team needs to leave plenty of room for teams to adjust the methodology to meet unique requirements. They should also solicit best practices from all purple teams, compile them into a body of knowledge, and provide regular opportunities for individuals and teams to share their experiences and lessons learned.

Templates. The best way to enforce standards and best practices is to bake them into development tools via templates, wizards, and default options. Here, the developer learns by doing. Tools that auto-handle slowly changing dimensions or different join types eliminate the need for analysts to write hundreds of lines of code, simplifying complex tasks. This reduces the likelihood of mistakes and rework and enables organizations to assign less experienced people to the purple teams, saving money. (See “Data Architecture as a Service: Liberation for Data Users). 

The best way to teach and enforce standards and best practices is to bake them into development tools via templates, wizards, and default options.

Data platform. Finally, the enterprise data team needs to establish a common toolset that simplifies and accelerates building and managing data solutions. At a minimum, the platform should include a cloud database for storing data; a data/ops tool for orchestrating code, pipelines, and tests; a data access management tool for governing data access; and a data observability tool for detecting data compliance and errors. The enterprise data team integrates these tools into a common data platform that abstracts the complexities of building data solutions. With a common data platform, purple teams can focus on applying their domain knowledge to a business solution rather than mastering data infrastructure, enabling them to work faster and more effectively.

What’s not listed as part of the common data platform are tools for building data applications, namely data transformation and data preparation tools and BI and analytics tools. Although it would be ideal if all purple teams used the same tools here, many domain teams that originate in the business already have a preferred tool and will fight to keep it. This is acceptable if they leverage the rest of the standard data platform. (See “What Can DataOps Do for You? Ask Roche”.)

Bottom-Up Flows – Red to Purple 

Business needs. Knowledge also flows from the business to the hybrid development team. The most obvious flow is the definition of business needs that form the basis of data-driven solutions. In most cases, the purple team facilitates this information exchange through formal requirements meetings and workshops. However, the best requirements are intuited by analysts or developers who permanently reside in the business. Because they sit with the business and know its people, processes, issues, problems, and opportunities at a deep level, they can proactively define solutions before the business even asks for them. 

Governance. Red teams are also responsible for data governance. Although business leaders would like to outsource governance to technical teams, only they can make decisions about the meaning and implications of data. For instance, businesspeople need to define metrics and dimensions, specify rules for data quality, define data privacy and security policies, and decide who can access what data.

The enterprise team can facilitate data governance by creating and staffing a program office that trains businesspeople how to govern data, facilitates governance decisions, documents results, and administers data governance tools and metadata. But the business must own the content, definitions, and policies and curate the data. They must take responsibility for these tasks, not technical people. Getting businesspeople to own data governance tasks can be a major bottleneck to development initiatives, which can’t proceed unless there is consensus about key data elements and data policies. 

Summary

Hybrid development teams are critical to the success of a data & analytics program. Data leaders must invest time, energy, and thought into the creation of these teams and how best to support them. It’s critical that they allocate staff time to nurture knowledge flows between the three groups. Without proper organizational design and sustained commitment and attention, the program will revert to centralized (IT-led) or decentralized (business-led) data initiatives that propagate data silos and data chaos. 

Our next article will drill into the composition of hybrid purple teams and blue enterprise data teams.