ABSTRACT: Let's reflect on the events of the past year and prognosticate on what may transpire in the months ahead.

At Eckerson Group, we love the end of the year. It gives us a chance to step aside from the daily cadence of creating data & analytics research and advising consulting clients. We can sit back and reflect on the events of the past year and prognosticate on what may transpire in the months ahead. 

Rather than just write about our predictions for 2024, we’ve decided to gather the team for a fireside chat (i.e., public webinar) where you can listen to our discussion and contribute your own thoughts and forecasts. Whether you read our formal predictions or not, please join us on December 19th from 2-3 p.m. Eastern for an informal conversation about what to expect in 2024.

Our predictions webinar will be a community discussion where we bring your thoughts and ideas into the conversation. Don’t be shy! Bring your eggnog or beverage of choice and celebrate the holidays and end of year with your favorite Eckerson research analysts and consultants. Register below!

Summary Predictions

Generative AI has certainly given everyone lots to talk and think about. The implications—both good and bad—are staggering. So, it’s no surprise that half our predictions involve GenAI, in particular, and machine learning and artificial intelligence (ML/AI) in general. 

Internally, we have feisty debates about GenAI and that’s reflected in our predictions. Dave Wells thinks it’s been overhyped and expectations will come back to earth in 2024 as organizations become more familiar with the strengths and drawbacks of the technology. Kevin Petrie is cautiously optimistic about GenAI and believes it’s a watershed moment that will deliver another paradigm shift in the way organizations harness data to achieve business objectives. 

Of course, there’s more to AI than GenAI. Our team believes that ML/AI will reshape nearly every commercial product, especially software products, such as data & analytics tools. Because of the pervasiveness of ML/AI, including GenAI, our team believes that organizations will need to pay greater attention to the data that feeds ML/AI models, improve AI governance processes, pay more attention to the environmental impacts of ML/AI processing, and heed the growing list of regulations governing the use of ML/AI.  

Outside of GenAI, our team contemplates the impact of the rise of mid-market data catalogs, data product platforms, and quantum computing. We also ponder whether ML/AI will usher in an era of data collaboration and interoperability (versus monolithic architectures) and whether data quality will evolve from manual data cleansing to automated remediation.  

What do you think will unfold in 2024? We’d love to hear your hopes, promises, and predictions. Until then, we at Eckerson Group wish you a wonderful and relaxing holiday season. We will see you bright and early in 2024! 

GENAI PREDICTIONS

1. GenAI Loses its Luster 

As the hype cycle for GenAI reaches the peak, reality and pragmatics begin to moderate the effect of new shiny-new-object enthusiasm. Practical leaders of data science applications begin to recognize the technology as something that is interesting, innovative, and a double-edged sword with potential for both positive and negative consequences. Sam Altman’s statement: “I probably trust the answers that come out of ChatGPT the least of anybody on earth” gets the attention it deserves. Policies, plans, and practices are put in place to guide the path to positive consequences and avoid the pitfalls of negative consequences. (Dave Wells)

2. Companies Get Serious About Data Governance for GenAI

Many companies piloted and deployed generative AI applications in 2023 and some even implemented domain-specific language models to enhance customer service, document-processing, and specialized research. But there's a catch: only 21% of respondents to a recent survey by Eckerson Group say they have sufficient data quality and governance controls in place now to support AI/ML initiatives, such as GenAI.  As a result, we predict chief data officers will double down on data governance to feed these new initiatives with accurate and trustworthy inputs, thereby reducing risks such as hallucinations, privacy breaches, and bias. They also will improve master data management and enrich the metadata they use to govern text files, the fuel for many language models. (Kevin Petrie)

3. GenAI Raises Interest in Green Data Management 

Training large language models takes a huge amount of data processing, which is once again raising environmental concerns. The environmental impact of data and data processing will be a common concern for most organizations as new sustainability goals are rolled out globally. Organizations will have more motivation to reduce data redundancy and have tighter disposition procedures to eliminate unneeded data at the end of its useful life. They will start to look more closely at the value of data and data processing activities to decrease their environmental footprint. (Sean Hewitt)

4. AI/ML Governance Goes Global 

Thanks to AI, we now inhabit a world where distinguishing fact from fake and vice-versa is becoming extremely difficult. As the euphoria around ChatGPT subsides, governments, nations and organizations will dive deeper into formulating realistic data & AI strategies based on both benefits and potential issues around hallucinations and fake data. Organizations will more closely attend to cybersecurity, governance, bias, ethical and regulatory issues as they embed ML and AI into the fabric of business operations. This is what Paul Daugherty, Chief Technology and Innovation Officer, Accenture has to say - “Ninety-five percent of executives say they believe in responsible AI. Only 6% have responsible AI programs in place in the company. So there’s a big gap.” (Sumit Pal)

5. AI Regulations Take Root

Europe has taken the lead with its AI Act passed recently after contentious debate about how to regulate generative AI and facial recognition. The tiered system of regulations that ranks AI applications based on risks that could impact safety or human rights is likely to be adopted by other countries and adhered to by most software vendors as a standard practice. The United States recently dipped its toe into the space when President Joe Biden issued an executive order to establish new standards for governing AI to ensure consumer safety, security and civil rights. The hope is that Congress will follow suit and pass legislation that will spare software vendors from adhering to a patchwork of state laws and regulations. (Josh Reid)

OTHER PREDICTIONS 

6. Organizations Get Serious about Data Collaboration 

Data integration patterns of the past — primarily data warehousing — struggle to meet needs of more recent data use cases for analytics, AI, and machine learning. While data integration continues to have value, it is limited value. Data management leaders are recognizing that data interoperability needs to become a core element of their data management architecture. Data semantics will have a central role to create clear, shared expectations for data content, context, and meaning. APIs, data products, data contracts, semantic mapping, open links, and data virtualization will be woven into data management architecture to enable interoperability. (Dave Wells) 

7. Cloud-Based Quantum Computing Comes of Age

This year we'll see cloud-based quantum computing make significant strides toward "quantum utility"—by applying quantum computing to solve real business problems, such as pharmaceutical design and materials engineering. Cloud-based quantum computing makes it feasible for more companies to apply quantum resources to their most processing intensive problems. There are two underlying developments that have hastened the arrival of quantum utility -- modularized quantum processors that can be strung together as needed to achieve as needed for more computing horsepower and new middleware that helps manage the integration of quantum resources with classic cloud computing resources. (Jay Piscioneri)

8. Automated Data Quality Remediation Debuts 

This is the year that automated data quality management achieves critical mass. Data Observability and Data Ops have laid the groundwork for identifying problems and managing workflows. Many of the vendors in this space are focusing on improving automated DQ remediation, which is the only way data teams can keep up with the volume, pace, and generally low quality of modern data.

In addition, the Data Observability and Data Quality market will coalesce and form something like DRE (Data Reliability Engineering) - on the same lines as Google introduced - Site Reliability Engineering. Data Observability by itself is too small and niche an area to be a market by itself. Data Observability will be integrated across every element of the modern data stack since it has become way too complicated - for any human to debug and perform root cause analysis. (Jay Piscioneri and Sumit Pal)

9. Mid-Market Data Catalogs Make Inroads on Incumbents

Mid-market data catalog vendors are taking page out of Clayton Christensen’s bestselling book, “The Innovator’s Dilemma” by offering faster, simpler, cheaper products than the incumbents, stealing market share at the low-end while adding functionality that will enable them to move up the competitive ladder. Mid-market products address a rather glaring pain point in the data catalog market: many customers have trouble gaining user adoption, making it hard to justify high price tags, which can run into the millions. Mid-market data catalog products don’t have the same functionality as the big boys, but they do have business-friendly interfaces, can be deployed within weeks, and offer transparent pricing that starts at less than $50,000. Expect these products to gain ground in 2024. Stay tuned for a 2024 Eckerson Group report titled “Mid-Market Data Catalogs: Leading Products You Should Know.” (Wayne Eckerson)

10. Data Product Platforms Materialize as a Market Segment 

As organizations embrace data products as a preferred method for delivering high-quality, reusable content to target groups, they recognize they lack a mechanism for data consumers to find, evaluate, and download data products. A slew of mostly small vendors has recognized the pain and are shipping “data product platforms” that facilitate the frictionless exchange of data products across organizational boundaries, both internal and external. Getting in on the act are data integration vendors, such as Informatica, Nexla, and RightData; data commerce vendors, such as Dawex, Harbr Data, Narrative.io, and Revelate; data analytics vendors, such as Coginiti and Promethium; master data vendors, such as Tamr; and data catalog vendors, such as Zeenea, ThinkData Works, and data.world. (Lyndsay Wise)