In times of beta, can an agile data strategy help deliver alpha?

Aabid Abbasi

EY Americas Financial Services Data and Analytics Principal 

Data and Analytics Leader in financial services. Converting data into insights. Avid cricketer. Global traveler. Father.

Chetan Saluja

EY Americas Financial Services Data and Analytics Senior Manager 

Passionate about leveraging data to drive growth, manage risk and build trust. Technology enthusiast. Foodie. Father.

16 minute read 15 Nov 2019

An agile approach to data discovery and insights is paramount for firms to align with ambitious plans and evolving markets at the same time.

We are living in the digital era. According to a new IDC spending guide, worldwide spending on digital transformation will be nearly US$2 trillion in 20221. Additionally, IDC estimates that by 2025, about 6 billion consumers (i.e., 75% of the world’s population) will interact with data every day, growing the so-called Global Data sphere to 175 ZB, with each connected person having nearly 4,900 digital transactions a day2.Data interactions of this kind are building the case for data as the new currency as it begins to reflect the end user’s digital DNA. Firms can now leverage insights from the data to target customers with personalized services and add value to existing ones.

However, the speed at which markets are evolving and will evolve in the future is exponential. This is where an agile approach to data discovery and insights is paramount for firms to align with ambitious plans and evolving markets at the same time. In the past, many firms adopted an agile approach but failed because the data tools and the firm culture were not mature. This is quickly changing. The few firms that are embracing new technology and pushing for the right culture are seeing notable success in this space. So, now is the time to establish an agile data strategy that is future-ready.

Firms will continue to explore enablers for an effective agile data strategy to gain and leverage strategic insights and outperform competition. As firms adopt this strategy to obtain such insights in short order, often times they may miss key considerations to manage risk. This is important as it can often present reputational risks for firms and in turn may lose customer trust. A well-rounded agile data strategy will account for such considerations while enhancing existing insights, generating value propositions and fuel customer growth.

In the next five years, firms with a data strategy that is innovative, embraces change and adds value with timely, intelligent insights will be the disrupters, not the disrupted.

Driving value in changing markets
(Chapter breaker)

Chapter 1

Driving value in changing markets

In the future, timely and actionable insights will differentiate firms from their competition.

Firms have and will continue to experience change in almost every sector. Those that embrace this change and drive valuable insights often emerge as market leaders. Actionable, timely and relevant insights that permeate throughout the organization are what allows the firms to lead with confidence in such volatile markets. However, getting valuable insights from data isn’t always easy. Some firms still follow the traditional data warehousing approach to insights, while leading firms are going agile to enable ongoing data exploration.

Historically, firms have relied on traditional data models to mine for insights; however, they remain inflexible, offer outdated insights and are difficult to maintain as data volumes grow.

This is changing quickly, and so are the market’s expectations. Firms are adopting an agile approach to data delivery and insights to stay ahead of the curve, constantly create customer value and preserve it. We at the EY organization believe that this paradigm shift will impact how firms utilize data on a day-to-day basis.

Experts across industries are closely monitoring firms’ efforts to achieve a delicate balance: increasing stakeholder value while lowering operational costs. The traditional approach to data exploration significantly drives up operating costs, with diminishing returns on investment. Agile data discovery and insights provide a lucrative alternative, as they leverage the advancements in both the technology and underlying support infrastructure to deliver high-value and cost-effective results. While the shift to agile may be a big cultural change for some firms, the benefits of agile data exploration far outweigh the cost of adoption, as outlined in Figure 1.

Figure 1: Comparing traditional and agile approaches

infographic about comparing traditional and agile approaches

As firms move toward agile, the focus is shifting from moon-shot projects to the curation of insights that are high-value, customer-centric and responsive to market needs. These firms are finding it easier to adapt their business processes to be more customer-centric and responsive to market needs. A good instance of this would be the success of the burgeoning FinTech population. These companies, often unencumbered by institutional costs, are increasingly dominating the market owing to specific product offerings and their focus on exceptional customer service. For example, financial institutions that moved quickly from static dashboards on their app to AI-infused personalized insights on cash flow, upcoming payments and curated offers based on interest, saw a noticeable spike in mobile app adoption and retention percentages.


Woman presenting ideas team
(Chapter breaker)

Chapter 2

Why is now the right time?

Transitioning to agile insights is now easier; the technology is here, and so are the frameworks.

FinTechs have disrupted the mortgage origination process, and traditional banks are already feeling the pressure to evolve or risk losing market share. On the other hand, regulatory requirements are evolving almost every year, in addition to the new ones with which financial services firms must comply. Most regulations already scrutinize how data is sourced, transformed and reported, while others, such as the EU General Data Protection Regulation (GDPR), restrict how personal data is processed and moved. Risk management is another area that has seen a tremendous appetite for agile: firms are leveraging third-party data and modeling it with internal data to better assess their overall risk appetite and profile.Such disruption in the market has strengthened the case for firms to adopt agile, and the time to do so is now.

In the future, firms will rely on real-time insights to unlock customer growth perspectives while navigating market volatility, geopolitical risks, regulation and disruption from niche players.
Aabid Abbasi
Principal, Data and Analytics, EY Americas Financial Services Organization, Ernst & Young LLP


So, why is now the right time? The adoption of agile for credible insights as the preferred choice isn’t new. Firms have not been able to realize the approach’s full potential because the underlying technology and infrastructure weren’t ready. But this has changed. The tools available in the market now cover the entire spectrum of the data ecosystem. From real-time data ingestion, to transformations, to the curation of real-time insights, the tools support agile development. Niche players are proof that a successful agile data discovery and insights life cycle is now possible. Tool features, core capabilities and industry solutions, as outlined in Figure 2, are helping firms realize the agile way of data discovery and insights.

Figure 2: Features, key capabilities and solutions enabling agile data discovery and insights

Key capabilities solutions enabling agile data discovery insights
women architects at work in the office
(Chapter breaker)

Chapter 3

Take note: key considerations often overlooked

Agile is successful when it has the right focus, skills and culture.

The need for agile leadership and preparedness has never been higher. Forty-seven percent of the participants in a recent EY poll said they were either passive or reactive in responding to disrupters.

  1. Prioritize high-value insights — Developing a giant data warehouse has worked well with high volumes of data but not so much with high veracities, velocities and varieties. The discovery of insights was also limited to a set of canned reports and fixed dashboards. Widespread disruption in the market and the need to innovate require firms to prioritize the highest-value value insights. A breakdown of work is required to develop such insights into timebound modules, and microservices capabilities or out-of-the-box industry solutions should be used to develop a high-value data discovery and insights pipeline in parallel.
  2. Culture, culture and culture — Developers must resist the urge to develop everything at once. The core business and technology teams need to collocate, brainstorm and work together in the development cycle. The teams should focus on a lean approach to the system of record, not Band-Aid fixes. The user community should dictate what data, insights and capabilities, such as self-service business intelligence (BI), are required. Solutions should be codeveloped with business-automated test cases that verify the integrity and quality of the data as it is ingested. Power users should also be part of the core team to help iterate through insights development. These users are an effective resource, as they are comfortable with tapping into both structured and unstructured sources and developing wire-frames required for analytics and insights. They are aware of the tool’s capabilities and are generally adept at augmenting artificial intelligence (AI) and machine-learning (ML) uses to generate high-value insights.
  3. Develop for data privacy and security — As data access becomes easier, data breaches become a big concern and regulators amp up compliance requirements, securing data has never been so important. While data privacy and security complement each other, both topics are top of mind for chief data officers and chief information officers. While agile promotes data democratization, embedding data privacy and security as part of every sprint is paramount. Firms are also establishing agile data governance committees that evaluate all data access aspects for every sprint cycle. Leading firms are leveraging test-driven development to execute automated data privacy and security checks for every change and also in production on an ongoing basis.
  4. Training and teaming — Often, it isn’t the direction the firm is taking that causes it to fail, it is the resources supporting the agile program. Talent who are or will be adept in tools that support real-time data ingestion, inbuilt data governance, and complex and customizable data visualizations should be trained or hired. In addition, they should understand when to leverage AI, robotic process automation (RPA) and ML. It is also important for the core team to understand the out-of-the-box industry solutions available in the market to make build-vs.-buy decisions. Success of such agile processes is hinged on resources working together as teams while moving away from an individual contributor mindset.
No system is uncrackable
(Chapter breaker)

Chapter 4

Agile data discovery and insights strategy

Successful agile data strategies will leverage both in-house and product-specific industry offerings while adopting the right level of governance.

Five key components of an effective data strategy are to ideate, assess, design, implement and govern.

Firms are realizing the value in shifting to agile for data discovery and insights as their digital counterparts evolve rapidly to dominate the market. Often, agile is misconstrued as an approach to development that involves shorter sprint cycles. This is a very myopic view. Driving value in evolving markets requires a holistic data strategy so that the benefits of agile development can be reaped.

An effective data strategy includes five key components: ideate, assess, design, implement and govern. A firm should overlay them with the key principles of agile and the possible external change factors to achieve a comprehensive framework that defines the organization’s data strategy. While many firms have a strategy similar to the one outlined in Figure 3, they often overlook governance and change with respect to the external factors.. In addition, teams seldom think of use cases outside the current user needs, which often backfires in these evolving markets. Many banks are so laser-focused on catching up with a competitor that they overlook important aspects of development outside their control. For example, a leading bank had to rewire its data ecosystem to comply with the GDPR — the talks of which have been going on for the past three years.

As you work through your strategy, focus on the new features that tools offer rather than creating them on your own. Product-specific vendor solutions are also a great way to kick-start your data strategy and catch up with your competition.

Figure 3: Agile data discovery and insights strategy

Agile data discovery and insights strategy
Woman Operating Digital Interface Technology
(Chapter breaker)

Chapter 5

Future of agile data discovery and insights

Insights platforms will continue to consolidate while expanding data perspectives and customer growth opportunities.

Agile methods for data exploration have been around for quite some time. With the right data strategy, firms can enhance their value proposition, increase customer satisfaction, achieve stronger compliance with regulators, reduce operational costs and, in the end, make intelligent decisions. The future of agile data exploration is promising, as it uncovers insights that were previously almost impossible or plausible only for select firms (see Figure 4).

Figure 4: Opportunities under an agile strategy

Infographics about opportunities under an agile strategy

Technology and business teams need to work together to leverage three key areas — tools, capabilities and product-specific industry solutions.

Forward-looking business users will be expected to know these areas and codevelop a compelling data strategy with technology partners.The creation of real-time, personalized and autogenerated insights from both structured and unstructured data will be made possible by tools and infrastructure that already support real-time ingestion from a variety of data sets with minimal coding. AI and ML with inbuilt support for various algorithms will quickly evolve to curate insights for intelligent decision-making. These technologies will also help firms consolidate businesses by providing a consolidated view of their portfolios with multiple cross-selling opportunities. For example, retail banking, wealth management and insurance customers will be served from a single app or portal, as the agile data strategy will simply involve a single view of customers’ data DNA.

With the evolution of the power-user role, product managers will be able to leverage the full power of self-service tools to quickly iterate through solutions with technology and support ad hoc insights needs. Technology teams will adopt a microservices approach to data to keep up with the speed of change and the need to evolve with it.

A lot of tools are providing access to prebuilt virtual reality (VR) data discovery and visualization libraries. This will change how users derive insights from data in real time. A leading financial services firm is in the final stages of viewing its data traffic using VR in real time to identify server slowness and take pre-emptive actions.

And finally, RPA will continue evolving to mine data from legacy systems and operational procedures, helping organizations gain insights from historical data.

rear view silhouette man kite surfer above sea sunset
(Chapter breaker)

Chapter 6

Key takeaways and next steps

Firms should engage with renewed focus on their agile data program to avoid disruption from their digital counterparts.

Although firms are already invested in and reaping the benefits of traditional data warehousing and BI capabilities, the market is shifting to data delivery tools capable of delivering results in a shorter time frame for real-time, data-driven decision-making. The headwinds to the traditional approach are only becoming stronger because of its diminishing value proposition. An agile data discovery and insights approach, along with the latest technology, will disrupt the way we obtain insights. Firms that aren’t agile yet should make the right investments and develop an agile data strategy in the next few months. Others should prioritize high-value insights and market needs, leveraging out-of-the-box solutions as well as core AI, ML and RPA capabilities.

Banking and capital markets continue to see greater challenges as customers want the best for less. Augmented investment decision-making, cashless transactions, micro-investing platforms, intelligent intraday liquidity, sentiment analysis and balanced trading decisions are among the examples of how firms can gain an edge over their competition in the future. Firms will combine these advancements with ongoing data privacy and trusted data programs to safeguard their success and position themselves as thoughtful leaders that provide credible insights. Also, the total cost of ownership and return on investment will be tangible, allowing firms to continuously evolve and focus on customer and market needs.

Data will become the No. 1 asset, currency, value proposition and digital fuel across all industries in the future. The opportunities are increasing, as are firms’ capabilities and willingness to realize the true power of data. Will firms realize the true potential of data? An agile approach to data discovery and insights holds the answer.

We would like to thank Alex Feld, Samhita Bandopadhyay and Shreya Tripathi for their contributions.

In times of beta, can an agile data strategy help deliver alpha?

Download (pdf)

  • Show article references

    1. Worldwide Digital Transformation Spending Guide, IDC, 2018
    2. The Digitization of the World: From Edge to Core, IDC, 2018
    3. “Three challenges for financial institutions as they compete with new market entrants, ” Tapestry Networks, 27 February 2019,, accessed [October 2019].




Firms that are embracing new technology and pushing for the right culture are seeing notable success in this space. Now is the time to establish an agile data strategy that is future-ready.

About this article

Aabid Abbasi

EY Americas Financial Services Data and Analytics Principal 

Data and Analytics Leader in financial services. Converting data into insights. Avid cricketer. Global traveler. Father.

Chetan Saluja

EY Americas Financial Services Data and Analytics Senior Manager 

Passionate about leveraging data to drive growth, manage risk and build trust. Technology enthusiast. Foodie. Father.