The U.S. Navy has made a big bet on the concept of “minimal manning”. It believes that a handful of cross-trained generalists are better equipped to operate a naval warship than a much larger number of highly trained specialists. Shortly, the U.S. Navy will launch 35 littoral combat ships (LCS), each manned by 40 cross-trained, multi-tasking sailors. This is a far cry from the 200 sailors required to operate a comparable “legacy” ship today and 350 during World War II. The new LCS ships, designed as modular trimarans, will cost less to operate and theoretically respond more nimbly to “fluid” conditions during combat. 

If the U.S. Navy has embraced flexible, cross-trained generalists to fight modern wars, should data analytic leaders do the same? Should we no longer hire people based on their past skills and knowledge but rather their innate ability to learn, solve problems, and collaborate with others? Is there a role for both specialists and generalists in a data development environment?

The Role of Spanners

I’ve argued in the past that the best solutions are developed by “spanners”, a term coined by Eric Colson, when he was head of data science and engineering at Netflix in 2010. Spanners are generalists who “span” the business intelligence (BI) stack—they analyze data, gather requirements, design data models, write ETL code, build BI dashboards, and write and execute all the tests—in essence, they are generalists who build an entire BI solution from beginning to end. 

No coordination costs. Spanners work more efficiently than an assembly line of specialists because there are no “coordination costs” where a specialist finishes her work, throws it over the wall to the next specialist, who picks up the task once they complete their current assignment. These repeated handoffs add up to significant delays over the course of a project. 

More context. Spanners also build better solutions because they have more context about what the business needs than a team of specialists who only see and build part of the solution. With a holistic perspective honed from working closely with the business for the duration of a project (often as a co-located resource), spanners continuously modify solutions to meet ever-changing business needs rather than cleave closely to a predefined set of requirements. 

Because spanners eliminate coordination costs and build better solutions, they deliver on the mantra of “better, faster, cheaper.” They are also more motivated than specialists since they have end-to-end responsibility for the project and are highly vested in its success. As a result, Colson claims spanners deliver 10 times the value of regular developers.

Not realistic. However, for most companies and teams, spanners are not a realistic option. There are too few people who have the skills, motivation, and interest to become proficient in a half-dozen tools and processes. And companies can’t afford them because they command twice the salary of normal developers. And spanners don’t scale. They work great in startups or departments in large companies but can create chaos and data silos if their changes are not systematically coordinated. 

Middle Ground? 

If a team of specialists is too slow and inflexible and a bunch of generalists too impractical, is there a middle ground? Early results from the Navy’s littoral ships provide some lessons. 

Sailors on the littoral ships are highly engaged and continuously learning new skills and roles. Most see this as a plus for their careers, and it keeps their jobs interesting. But the Navy has experienced hiccups with the early rollout of the ships. Without a crane expert on board, one LCS had to summon an expert and wait four days for him to arrive. The crew of another LCS failed to oil a main engine gear, forcing a $23 million repair. And the crew of another LCS put a seal in the wrong hole, flooding its engine with seawater. 

For the Navy, these problems highlight the need for experts. But this doesn’t mean experts have to carry out the tasks in all cases, but they need to be close by, either to train or supervise the generalists or fix a complex problem. And generalists are not ideal in some situations. For example, the Navy now realizes LCS are unsuitable for forward combat because there are not enough people on board to stand watch. So the Navy is learning how to balance the need for experts with a ship run by generalists.

Rules of Thumb

So how do the experiences of the U.S. Navy apply to our world of data analytics? What have we learned about how to optimize the development and delivery of BI solutions? Here are three rules of thumb to consider. 

Rule of Thumb #1 - Specialists at corporate, generalists in the business. The corporate BI team (or data analytics team) should house specialists in data architecture, data modeling, data integration, data quality, and data science. Their job is fourfold: 1) build enterprise or complex BI solutions that no department will build or fund 2) provide the first line of support for data and analytics specialists in the field, 3) convert prototypes from the field into production applications, and 4) maintain a knowledgebase of best practices and technical components. 

Conversely, every department and line of business should have its own data and analytics practitioners. In common parlance, we call them “data analysts” or “data scientists”. These folks are business people first, and technologists second. They use data and analytics technology to answer ad hoc business questions, build local reports, and create predictive models. Like a spanner, they do a bit of everything fairly well, but nothing extremely well. When they get in over their heads, they need to consult a specialist. 

Besides building local solutions, these generalists are the “eyes and ears” of the corporate team in the business units. They can accelerate requirements gathering for complex or enterprise projects, evangelize the eventual solution, and help ensure its adoption. This is why it’s very important for organizations to have a BI program (or BI Competency Center) where corporate specialists and field generalists know each other and work together.

Rule of Thumb #2 – Teams are better than individuals. Agile taught us the value of self-organizing teams and time-boxed development. But it remained silent on whether to staff the teams with specialists or generalists. It turns out that you need both. Agile teams need specialists with the right skills to solve problems quickly and efficiently, but if the specialists are sick or on vacation, the agile team grinds to a halt. Also, since specialists can only do one thing, they will spend a lot of time on an agile team doing nothing. Since that’s not very efficient, some agile teams keep specialists on call instead of making them full-time members of the team. But this taints the agile methodology with assembly-line techniques, creating a hybrid approach that cynics call “Scrumerfall”.  

Many agile leaders now recognize the importance of creating cross-trained development teams that work in an agile manner. Each member of the team brings their specialty to a project, but they are also responsible for learning the skills of every other team member. This is both daunting and invigorating. For example, it’s daunting for a BI developer to build data models, but it’s invigorating to learn those skills under the close watch of mentors on the team. This creates a stronger team, comprised of full-time members, who can fill in for each other. Along the way, they develop “fluid thinking”—just like the LCS sailors—and are more apt to devise novel approaches to new and unexpected issues that inevitably arise in each project. 

Rule of Thumb #3 – DataOps required. Any development these days requires a disciplined, methodical approach that balances speed and quality. DataOps enables development teams to scale in size without eroding the quality and quantity of their output, which is what usually happens as teams get bigger. To achieve breakthrough results, DataOps teams continually wrestle with how best to balance the capabilities of specialists and generalists as requirements and technologies change. (See our latest report, “Best Practices in DataOps: How to Create Robust, Automated Data Pipelines.”)

Summary

To optimize your technical resources, hire specialists for your corporate data analytics team and generalists in the field who work side by side with business users. Create a BICC to ensure these groups work closely together, sharing best practices and providing mutual support. When you need a team to build a solution, create an agile self-organizing team that cross-trains its members. This creates an ideal mix of specialists and generalists who can innovate on the fly and deliver optimal solutions.  

The above rules of thumb are not hard and fast. Every team has to learn and adjust as it goes. The U.S. Navy jumped into “minimal manning” with its littoral ships, perhaps without doing sufficient testing, but it’s now learning from some of the unforeseen consequences. The key is to reserve time to evaluate your own processes and ways to improve them. This type of continuous learning lies at the core of DataOps and makes it a powerful tool for transforming your data teams into high flyers.