Eckerson Group believes in holding itself accountable. Every year we give you predictions for the next, but we also give you a scorecard that shows how we did last time. We made 10 predictions for 2021. Here’s how we did:

Prediction #1: Work-Life Balance

Last year, Wayne both wished and predicted that, post-pandemic, our newfound appreciation for life outside work would continue and that we would prioritize work-life balance going forward. Regrettably, we are still very much in the pandemic. Nevertheless, 2021 was a banner year for reclaiming work-life balance. Beginning in April when more than 4 million people quit their jobs, “The Great Resignation” has defined much of our late pandemic experience. Workers have left their positions in droves, unwilling to sacrifice their physical and mental well-being. At the same time, job seekers are prioritizing work-life balance as they look for positions. A new study found that work-life balance is now more important than salary or benefits.  On the employer side, the extension of the pandemic and universal staffing shortages have made businesses more amenable to these demands, and many companies have now accepted the inevitability of long-term flexible work-from-home arrangements.

Prediction #2: AI Marketplaces Drive AI Adoption and Monetization

AI marketplaces never materialized. In fact, there seems to be little appetite for them at all. Between open machine learning (ML) libraries and the broader commoditization of ML models, marketplaces for proprietary algorithms seem like a non-starter.

Prediction #3: Intelligent Process Automation (IPA) Takes Root

The trend of intelligent process automation (IPA) continued this year with UIPath, Blue Prism, and Automation Anywhere all leveraging artificial intelligence to enhance their traditional robotic process automation (RPA) solutions. That said, Andrew’s prediction that low-code vendors Appian and OutSystems would integrate or partner with classic RPA vendors didn’t come to fruition. For the moment, they continue to go it alone.

Prediction #4: Privacy Management Becomes Embedded in Analytics Tools and Data Pipelines

While privacy persisted as a major concern for businesses over the last year and vendors continued to promote features for governed pipeline development, there were no major events or announcements in the space. This is a long term trend, but not one that saw dramatic acceleration in 2021.

Prediction #5: Data Ethics Becomes a Data Governance Imperative

In the past year there’s been progress on data ethics, but the pace is slow. On the bright side, the Enterprise Data Management Council (EDM Council)  incorporated data ethics into their Data Capabilities Assessment Model (DCAM) and the availability of data ethics education has increased thanks to groups including universities, the EDM Council, dataethics.net, eLearningCurve, moderngov.com, and other training organizations. In addition, the Data Ethics Framework published at data.gov is now part of the Federal Data Strategy in the President’s Management Agenda. Regrettably, when it comes to the private sector, data ethics remains absent from many lists of CDO priorities. Instead, the emphasis is on monetization, as well as encouraging data sharing and collaboration.

Prediction #6 - More Companies Hire Full-Time DataOps Engineers

This month, there were three times as many positions available for a “DataOps Engineer” on job sites such as LinkedIn and Glassdoor than this time in 2020. In addition, the companies hiring for these roles were no longer just big tech companies and large consultancies. 500-person real estate groups, transportation companies, and waste management businesses all had live postings for DataOps Engineers.

Prediction #7 - Cloud Data Platforms Swallow the Data Warehouse and Data Lake

There’s no doubt that data warehouses and data lakes still exist, but, as Kevin predicted last year, the cloud has seen major consolidation. Snowflake, once the standard bearer of the cloud data warehouse, no longer identifies as such and Databricks, a leader in cloud data lakes, now pursues the data warehouse market.

Prediction #8 - Quality Standards Begin to Emerge for Machine Learning (ML)

As Dave predicted, the International Standards Organization (ISO) took up the question of quality standards for analytics and machine learning models this year. Currently, there are four projects in preparatory phase: ISO/IEC AWI 5259-1, ISO/IEC AWI 5259-2, ISO/IEC AWI 5259-3, and ISO/IEC AWI 5259-4. This puts us right on track for Dave’s suggestion that the ISO standards would publish in 2024 or 2025.

Prediction #9 - Companies Invest in Data Governance Technologies, Staff, and Certification Programs

Companies continued to invest in data governance this year with the adoption of new tools such as data catalogues growing steadily. They also hired more dedicated staff to tackle governance challenges. Data governance analysts, data stewards, and data curators remained in demand throughout 2021. That said, there was no watershed moment, just the furtherance of a multi-year trend.

Prediction #10 - Data Pipeline Observability Goes Mainstream

Data pipeline observability gained steam last year, but remains in its early days. Recognition of the need for observability continues to grow, as enterprises worry about both performance and data quality for analytics workloads. One observability start-up, Monte Carlo, raised more than $60 million this year to pursue its vision, while its competitors Acceldata and Bigeye raised another $30 million and $17 million, respectively.

Takeaway:

By and large, Eckerson Group was effective last year at predicting general trends. Only one out of our ten predictions appears to have been wholly off base in retrospect. Not too shabby! We did anticipate more significant paradigm shifts in the world of data governance than materialized, but much of the overprediction can be attributed to the prolongation of the Covid pandemic. Like many others, we hoped that widespread vaccinations would lead to a more or less normal environment this fall, giving organizations the stability needed to make big changes and investments. The Delta surge unfortunately put an end to that optimistic outlook. Given the continued economic instability and the difficulty of making long term business projections in the current environment, it’s unsurprising that most organizations opted for steady progress rather than dramatic change on many of their data governance initiatives.

Let us know in the comments if you have any additional reflections to share on the year gone by, and be sure to check out our new predictions for 2022!