What’s Driving the Data Prep Market? TDWI Digs In

(faithie/Shutterstock)

It’s no secret in the analytics community that data quality is a problem. But how big a problem is it? TDWI, a respected provider of data warehousing insights, shed some light on the topic with a new report.

In the “best practices” report, titled “Improving Data Preparation for Business Analytics,” senior TDWI research director David Stodder and his team sought to dig into the market for self-service data preparation, and reveal the core drivers powering a market that Gartner predicts will be worth $1 billion in three short years, growing from a 5% penetration rate to more than 10%.

Along the way, TDWI found answers to several questions, including: how much time data pros are spending cleaning data? To what extent do data pros rely on the IT department? And what’s the overall maturity level of data prep processes?

The results of the survey of more than 400 data professionals show that, while organizations seem to be aware of the scope of the problem surrounding data prep, the situation does not seem to be improving much. To wit:

• According to the TDWI report, manual data preparation continues to burden analyst resources. The report found that a majority of survey respondents said analysts spend 61% to 80% of their time manually prepping data.

• 86 percent of survey respondents reported that it’s “very important” or “somewhat important” to reduce the amount of time and resources spent on data prep for analytics and BI.

• Nearly 50% of analysts report relying on their IT department for the first step of data prep tasks, which can take anywhere from two days to four weeks to complete. Many analysts reported that they received the data too late to have a business impact, the report says

• Only 4% said their data preparation tasks have been entirely automated through products, while a majority say they prepare data by hand every time.

Prakash Nanduri, CEO and Co-founder of self-service data prep software provider Paxata, which co-sponsored the TDWI report, said it’s not surprising to hear that people are dissatisfied with their ability to find relevant data and understand how to use it appropriately for BI and analytics.

“While companies are drowning in data, business teams are thirsting for information they can actually use,” Nanduri says in a statement. “This important research indicates that the strong demand we have seen in the last three years for our enterprise-grade self-service data preparation platform is just the tip of the iceberg.”

But there’s hope yet for the future, Stodder writes in the report’s intro. “With improved practices and technologies for data preparation, organizations can better deal with current data troubles and prepare for future challenges arising from new data and user requirements,” he writes. “Innovation in data preparation could not be coming at a better time.”

The 44-page report, which you can download from Paxata’s website, was also co-sponsored by a several other prominent members of the data prep market, including: Alteryx, Attivio, Datameer, Looker, Pentaho, RedPoint Global, SAP/Intel, SAS, Talend, Trifacta, Trillium Software, and Waterline Data.

Related Item:

Why Self-Service Prep Is a Killer App for Big Data

Why Big Data Prep Is Booming