Why data lakes are underutilized in supply chains and how to fix it

Despite the potential of data lakes, the same research reveals that 17% of companies admit they are not using them effectively, essentially stating that their data lakes haven’t been particularly helpful in advancing supply chain decision-making. Additionally, a further 6% of those surveyed say they do not have a data lake yet but plan to implement one. Similarly, another supply chain research survey by the EY organization shows that most supply chain and operations executives (73%) are planning to deploy generative AI (GenAI). However, 62% of those respondents have reassessed their projects, and only 7% have completed the implementation.

Overall, both surveys reveal that even though many organizations have moved ahead with data lakes and tools such as GenAI to improve their supply chains, they still have a long way to go, especially as they journey towards autonomous supply chains.

While data lakes offer potential for significant insights, organizations need to address several challenges to fully realize their benefits: 

1. Unlocking business value from data: Enterprises gather significant amounts of data from their supply chains, but the question is, are they driving significant value from that data? One of the overarching problems many companies face is that data lakes were not built on a unified supply chain data model, data is often siloed, with manufacturing data in one repository, planning, inventory and logistics data in another, and customer service and supplier data in yet another. This issue persists even when the data lake is developed on the same vendor platform, as the integration and interoperability challenges remain. Efforts to unify these data structures requires effort across 12-24 months — a timeframe that organizations generally cannot afford.
   
   
2. Visualization versus analytics for future planning: Another major issue is that most of the data captured in data lakes is primarily used for visualization dashboards, which provide a rear-view key performance indicator (KPI) that shows executives what happened yesterday, last month or last year. Supply chain decisions are cross-functional and as companies navigate volatile markets, we see organizations struggle to use AI capabilities in forecasting, dynamic inventory management, logistics optimization and more. Many large companies do not have these analytics capabilities or are in the early stages of implementing them.
   
   
3. Building trust in data: Lastly, few organizations completely trust their data. Customer data is sometimes not consistent. As a result, linking customer records across the company can be difficult, inconsistent product data may lead to errors and data is not often synchronized across systems in a timely manner. And because of data silos, most organizations have multiple dashboards consisting of their own functional data. This makes it difficult to decide whose version of the “truth” to use for across the enterprise analytics. With the understanding that data is never perfect, organizations must address the challenge of how to improve trust regarding their data. 

To that end, organizations should consider a solution that looks past the common visualization dashboard to a forward-looking model that enables the organization to run different simulations to test different scenarios. GenAI offers the potential to be a significant game-changer in this effort, better enabling organizations to tackle large amounts of data and make more informed and intelligent supply chain decisions using natural language interfaces to query their data. GenAI also has the potential to enable organizations to run multiple end-to-end supply chain simulations that will explore potential options for optimization, both in manufacturing and overall coordination of logistics.

Organizations often mistakenly treat supply chain simulations as linear processes, believing they can only begin utilizing them once their data and systems are fully ready. The reality is that data is never fully clean and ready for business use; it is a continuous and evolving process. Moving forward, supply chain leaders should consider adopting a new approach when transforming their data lakes and AI capabilities.

This new approach entails viewing data lakes as valuable assets that allow decision-makers to rely on insights through new experience-based tools and technologies. This means organizations should stop building data lakes merely as dumping grounds for storing information and instead treat them as tools that – when combined with data analytics – can help optimize the supply chain and other parts of the operation.

At this stage, the organization should add an AI layer that leverages ML to analyze data with a clear eye on a wide range of use cases. GenAI then serves as the top layer, and we see many organizations that deploy GenAI are beginning to transform the way they interact with data, running simulations and other scenarios that provide a forward-looking view of the supply chain ecosystem.

One critical consideration all organizations will need to make as they build out their ability to tap GenAI and leverage it for broader decisions and deploy a truly autonomous supply chain is to establish clear governance and accountability structures. This includes defining who will take ownership of the AI models and ensuring there are protocols in place for monitoring and addressing any issues that arise. Accountability is crucial here. Organizations cannot simply blame the AI model if something goes wrong; there must be explainability and an audit trail for tracking the decision-making process of the AI model, and human oversight to manage and rectify any potential problems.

A second consideration is the incorporation of real-time data streams. To fully harness the potential of GenAI for simulations, organizations must ensure that their data lakes and AI models can ingest and process real-time data from various sources, including internet of things (IoT) devices, sensors and external data feeds. This capability will allow companies to respond swiftly to fluctuations in demand, supply chain disruptions and other dynamic factors impacting the supply chain.