BI’s Inflection Point: The New Fast Data Exploration Mandate

Mikko Lemola/Shutterstock.com

Intel (INTC)’s Andy Grove famously wrote and spoke about the difference between ordinary change and a strategic inflection point (SIP) with significant impact to the health and survival of an organization. He defined strategic inflection points by the magnitude of impact on a business, quantifying them mathematically as a 10X change that the business has been accustomed to. Grove also noted that SIPs aren’t only technology driven, but can be precipitated by new or shifting competition, new channels of distribution, social and cultural shifts, regulatory changes, and so on.

In his book, Only the Paranoid Survive, Grove wrote: “We managers like to talk about change so much so that embracing change has become a cliché of business. But a strategic inflection point is not just any change. It compares to change like Class VI rapids on a river, the kind of deadly and turbulent rapids that even professional rafters approach gingerly, compare to ordinary waters.”

Enterprise businesses of all kinds now face a strategic inflection point and sense of urgency when it comes to their ability to harness more sources of data quickly and explore it at scale to uncover business-critical insights. They need to make it accessible to people across the organization so they can quickly explore it, analyze, iterate on it, and then collaborate and act on key insights with internal stakeholders, external partners and customers. And they need to do so, almost daily, hourly, and on a fast cycle with the freshest and most recent data visible, not data that represents past events. This is resulting in the complete reinvention and redefinition of Business Intelligence and a strategic inflection point around data and data exploration in organizations globally.

The New BI Imperative

Most areas of enterprise software are reinvented and redefined every 7-10 years. In most cases when reinvention has occurred, it was an option, not an imperative. For instance, when server virtualization was popularized by VMware, it was an option; otherwise we’d have continued with existing physical-only server scaling. When ServiceNow redefined ITSM, it was an option, not an imperative. When Box redefined secure file storage in the cloud for enterprises, it was an option.

After decades of little reinvention in the area of Business Intelligence, we’ve reached an inflection point that makes the modernization of BI an imperative today, not an option. The classic BI approach has not evolved much outside of prettier visualizations, but in the last 2-3 years it is suddenly undergoing rapid change. It’s being replaced by a new BI model that leaps it ahead into the 21^st century in terms of faster time to insights and ability to see “what’s happening now,” ask new questions, and get near real-time answers to steer the business in hyper-competitive market environments.

Both the data landscape context that we live and work in and business competition are in a state that we’ve never seen or experienced before. Add to that, fundamental technology advances that have occurred in cloud, user interfaces, and data processing, and together, we are now seeing a kind of runaway acceleration in BI innovation that surpasses a lack of change in approach from previous decades.

A Perfect Storm for BI Modernization

As a result of the data environment, business environment, and core technology acceleration, Business Intelligence is being completely transformed for the modern age. The transformation applies to everything; from the way data is accessed, how it’s prepped, how it’s blended, how fast it happens, how easy it is, and how it’s freely explored, analyzed and collaborated on.

Key factors have converged across three core areas – technology advances, environment, and modern business needs – that create a perfect storm that urgently requires organizations of all kinds to make a leap up to modern BI so they can succeed and thrive in an age of data transformation for business.

Technology Advances

1. To address the need for scalable and fast data processing, Apache Spark has spiraled into the forefront as a catalyst and industry change agent, replacing Hadoop MapReduce. Frameworks like MapReduce weren’t designed for anything approaching near real-time analytics on large data volumes. It was fine for batch processing but too slow and complex to be viable for modern data needs. Spark is now being broadly adopted because it provides a robust, massively parallel cluster-computing platform that runs programs 100X faster than MapReduce in-memory and 10X faster on disk.

2. There’s also a significant shift in cloud adoption as more enterprises have moved or plan to move data processing and collaboration from on-premise to the cloud. The attractive economics and speed advantages of cloud-hosted solutions are further accelerating the shift to more modern architectures.

3. More machine intelligence is being developed and engineered into modern data analytics platforms, which eases the cost and complexity of manual data modeling, addressing a lack of data scientists and those skilled in coding and data analytics.

Environmental Factors

1. We’ve experienced an unprecedented explosion in data volume, variety and complexity with data volume forecasts to explode by 4,300 percent by 2020 ¹ from a study by CSC. The proliferation of new channels for data creation and consumption across mobile and social platforms, and the emergence of the Internet of Everything, are further fueling this huge growth.

2. The rise in information workers is expected to increase to 865 million people by the end of next year, according to analyst firm Forrester Research. More business users than ever will access, analyze and share data-driven insights across industry sectors.

3. Data governance of who gets access to what data is rising in importance as more sensitive data becomes accessible across more channels than ever before. Data security is also a top concern with the increase in security breaches, hackers, and the continued rise of a global cyber-security syndicate.

New Business Requirements

1. Immediate Insights on What Is Happening Now: There’s an urgent need to access more data and speed business insights. For example, gaining insights on customers and their motivations are top priorities for CPG brand executives. A better understanding on responses to in-store product placements and consumer uptake, or online offers and online versus in-store purchasing swings, or immediate insights on “what’s happening” with customers hour by hour so one can act to capitalize on consumer trends, all lead to a deeper knowledge of the customer and ultimately faster top line growth. Similarly, in the data-driven healthcare and pharmaceutical industries, faster and more holistic insights can speed the cycle from diagnosis to cures, or provide just-in-time data on systemic patterns to alert hospital caregivers of any diagnosis of life-threatening conditions. These early-warning systems can not only save lives, but also reduce hospitals’ costs through savings in medication, staffing and other costs.

2. Fast Discovery and Blending of More Data Sources: Two-thirds (64 percent) of companies are already trying to combine five to 15 sources of data² in their quest to find richer, more meaningful insights and additionally, break the silos of data repositories that have emerged in their legacy data infrastructure over time. Whether combining data to see more answer new questions or combining data to overcome legacy silos, there is a massive need to speed and ease disparate data blending. If business users are trying to do it themselves, a task that has become an impossible feat, shockingly, 92 percent of these business users are still using Excel for analysis for lack of better alternatives.² This situation is simply not sustainable.
3. Business Self-Reliance: Through 2017, Gartner predicts that the number of business users doing data analysis will grow five times faster than the number of highly skilled data scientists ³ and most business users and analysts will access self-service tools to prepare data for analysis.⁴ There’s already a broad market shift happening for everyday business people, versus data scientists or analysts, to access and blend disparate data and become more self-reliant in reaching deeper insights. This shift requires the development and adoption of easy-to-use, self-service, simpler data analysis solutions.

Today, the modernization of BI and data analytics is experiencing its fastest pace of change ever at a strategic inflection point in an age of rapid data transformation. BI change has become an integral part of every organization’s quest to empower business users to be self-reliant in accessing, discovering, and exploring more data and more insights. We are now reaching a market inflection point where rapid BI transformation is happening. Organizations of all kinds in quick moving, highly competitive sectors such as CPG, national retail, healthcare and logistics/manufacturing urgently need near real-time, immediate, business-ready insights that cross more data sources, to survive and thrive in today’s hyper-competitive marketplace.

¹ CSC “Data Universe Explosion and the Growth of Big Data” – 2012

2 Harvard Business Review, Analytics Services Report: “Data Blending: A Powerful Method for Faster, Easier Decisions” – August 31, 2015

³ Gartner, Inc. Report: “Smart Data Discovery Will Enable a New Class of Citizen Data Scientist” – June 29, 2015

⁴ Gartner, Inc. Report: “The Rise of Data Discovery Has Set the Stage for a Major Strategic Shift in the BI and Analytics Platform Market” – June 15, 2015

About the author: Sharmila Mulligan is the CEO and founder of ClearStory Data, a big data analytics startup based in Menlo Park, California. Sharmila
has spent 18-plus years building game-changing software companies in a variety of markets. She has been EVP and CMO at numerous software companies, including Netscape, Kiva Software, AOL, Opsware, and Aster Data. She drove the creation of several multi-billion dollar market categories, including application servers, data center automation, and big data analytics. She is on the board of several tech companies based out of Silicon Valley, adviser to numerous companies, large and small, and an active investor in early-stage companies.