More Companies Walking the Big Data Walk

(Thomas Zsebok/Shutterstock.com)

There’s a big difference between saying you’re going to do something, and actually doing it. That dynamic has been at play in the big data arena for some time now, as evidenced by a lot of talk, but a relatively low level of action. But now it appears that the number of companies embarking on big data and advanced analytics projects is nearing a critical mass.

A recent survey by CompTIA indicates that the fraction of companies embarking upon big data initiatives continues to accelerate. In its 2015 Big Data Insights and Opportunities study, the research organization found that 51 percent of survey respondents report having big data projects in place today, up from 42 percent in 2013. In a corresponding shift, just 36 percent report having a big data project in the planning stage, down from 46 percent two years ago. (The survey is based on responses by 402 business professionals, and has a sampling error of 5 percent.)

“While there may be some amount of ‘big data washing’ going on, the net result is that companies are moving forward with data initiatives and seeing results,” CompTIA says in its report. “A net 72 percent of companies feel that their big data projects have exceeded expectations.”

That rosy figure is backed by other surveys, including Forrester‘s 2015 report on data and analytics. That report found that the percent of companies currently using predictive analytics technologies jumped from 31 percent in 2014 to 48 percent in 2015. Similar jumps were seen in other areas, including location analytics, text analytics, advanced visualizations, streaming analytics, and search/interactive discovery.

“It’s interesting to see the momentum and how enterprises are talking about what technologies they’re using,” Forrester analyst Mike Gualtieri said in a recent keynote at the Spark Summit East conference. “There’s a lot of at least reported momentum for enterprise in using these various forms of advanced analytics.”

This information is about as good as it gets, short of barging into a thousand headquarters of American corporations and ripping apart their IT systems, which would earn the earnest reporter a quick trip to the pokey. While both Forrester and CompTIA qualify their survey results to give themselves some wiggle room in case these self-reported numbers turn out to be phony, that would appear to be unnecessary, if you accept that the incidence of “big data washing” is relatively constant and isn’t likely to vary much from year to year.

This groundswell of movement in big data analytics is being felt up and down the supply chain. Adam Wilson, CEO of Trifacta, has seen increased interest in his big data tool, which simplifies the tedious data preparation activates that must be completed before meaningful analysis can be performed.

Hadoop cluster sizes are growing, according to AtScale’s Hadoop Maturity survey

“In the past a lot of these got started as science experiments with IT organizations,” Wilson said in a recent interview with Datanami. “As we’ve seen increased maturity in the market, there’s much more focus on driving real business value and ensuring that you can demonstrate the impact you’re having on decision making.”

Many, if not most, of Trifacta’s customers are seeking to get a handle on massive amounts of unstructured and semi-structured data that’s being stored in Hadoop, which has become the de-facto standard for next-generation data-oriented operating systems. As such, Hadoop adoption is a proxy for adoption of the wider big data analytics ecosystem.

Forrester has stepped out on a limb by predicting that every major company will eventually use Hadoop. We’re not there yet, but a recent TDWI survey shows that Hadoop is well on its way. The survey, which was conducted in 2014, found that that the number of Hadoop clusters in production grew 60 percent in two years.

“At this rate, 60 percent of users surveyed will have Hadoop in production by 2016, which is a giant step forward,” the TDWI analysts, Philip Russom and Fern Halper, say in their 2015-2016 “TDWI Hadoop Readiness Guide,” which the company released in November.

Source: CompTIA survey

Big data analytics is clearly becoming an investment priority for many companies. As the volume of data thrown off by a myriad of sources continues to grow unabated, the momentum behind doing analytics upon it is growing, while the arguments against doing something is shrinking.

The sheer growth in data appears to be a big factor driving this shift. Recent data from network and IT equipment supplier Cisco shows the volume of data generated on the Internet by mobile devices like smartphones, tablets, PCs, and TVs is slated to surpass 1 zettabyte of data in 2016 by the first time, growing to 2 ZB by 2019. That’s gone beyond big data to mind-boggling data.

While the data is there and the analytic tooling is getting easier to use all the time, there are still substantial obstacles preventing companies from achieving their big data dreams.  The shortage of highly skilled data scientists who can identify insights buried in gigantic data sets is a real concern for the industry.

Another concern is how best to make use of those insights. You can have a team of the best data scientists and analysts in the world generating great insights, but if they aren’t put into action within the confines of a business model, then all the investments have gone for naught.

This was the conclusion of a recent survey conducted by Square Root, which found that nearly one-third of companies failed to take meaningful action as a result of their analytics initiatives. “Simply chasing down large volumes of information can lead to decision paralysis and waste both time and money,” Square Root CEO Chris Taylor said.

These are great times to be in the big data analytics business, to be sure. But as the data volumes grow and spur the creation of many more analytics projects, the pressure on executives to deliver results–to not just talk the data analytics talk, but to walk the walk–has never been greater.

Related Items:

Another Survey Questions Value of Big Data

Hadoop’s Second Decade: Where Do We Go From Here?

What Does 2016 Mean for Data Science?