Three Ways to Close Your Company’s Data Science Skills Gap Now

(LeonidKos/Shutterstock)

It should come as no surprise that demand for folks with data science expertise exceeds supply. In fact, according to some McKinsey, there are only half as many qualified data scientists as needed. The good news is the market will likely resolve the shortage in the long run. But in the short run, the talent gap creates some challenges for an organization looking to get ahead with data.

Thanks to their ability to use math and computer science to turn big data into business gold, data scientists are the rock stars of the advanced analytics world (at least as data scientists have traditionally been defined). As more companies start investing in AI, they’ve looked to data scientists to lead the way.

In response to the increased demand for data scientists and other folks at the sharp end of the data science stick – we’re looking at you, data, machine learning, and deep learning engineers —  it’s driven their salaries up into the mid six-figure range. Uber pays its AI engineers an average of nearly $315,000, according to one report, with many other big firms in Silicon Valley and elsewhere paying more than $200,000 per year. Obviously, this is a great time to be a data scientist (or even “research scientist,” as some data scientists have begun calling themselves).

But there’s a big problem with the data scientist supply chain: there’s just not enough of them. While universities have pivoted sharply to data science by adding new degree programs and curriculums, it has barely put a dent in demand. A LinkedIn report from August found more than 151,000 job postings for data scientists, with acute shortages being felt in big tech hubs like San Francisco, New York City, and Los Angeles.

The gap between data scientist supply and demand is big (Graphic courtesy of BHEF report)

Some compelling statistics about the current data talent situation were laid out in Correlation One‘s “Future of Data Talent” report, which was published today. The report quotes a Harvard Business Review study that found 40% of companies claim they’re “unable to hire or retain data talent due to a lack of supply.”

It also found there will be 2.7 million new data-related job postings in the U.S. by next year (a figure that originated with a PwC study commissioned by the Business-Higher Education Forum). That figure corresponds with an Asia-Pacific Economic Cooperation study that found a 20% increase in demand for data talent by 2020.

“Skilled data professionals are the backbone of any data initiative, yet most companies struggle to identify and hire skilled data talent,” Correlation One’s co-founders and co-CEOs, Sham Mustafa and Rasheed Sabar, write in the report. “Companies need data scientists, data engineers, quantitative researchers, and machine-learning specialists. And they need data analysts, product managers, database administrators, and business intelligence analysts with data literacy.”

Go Down Market

One potential way that companies can address the problem is to stop looking exclusively at top-tier universities for data talent, says Correlation One, a company that helps companies find data scientists.

While excellent data science prospects can be recruited out of Ivy League schools like Harvard and Yale and top-tier public schools like NYU and UC Berkeley, with the acute shortage of data scientists and the ridiculously high salaries, companies would do well by themselves to consider graduates from a range of other schools.

(Graphic courtesy of Correlation One’s “Future of Data Talent” report)

This recommendation is born out of empiricism: Correlation One actually tested more than 50,000 students from more than 200 universities to gauge their data science aptitude. The company concluded that excellent data science candidates can be found in places like University of Michigan, University of Illinois, the University of Texas, and UCLA.

“While on average there are more elite data science and analytics students at tier one schools, by volume, there are significantly more elite students at tier two and tier three schools,” the company writes. “Most organizations cannot afford to outbid tech giants, and instead, settle for average students from tier one schools. In doing so, they miss out on elite students from tier two and tier three schools, who are not only more skilled but also cheaper and easier to hire.”

Companies should also look beyond grades or test scores as the sole indicator of a data scientist’s future value, Correlation One says. “Undervalued talent in schools like Baruch College [part of the City University of New York network] have candidates in their top 10% that out-perform average tier-one school candidates,” the company says.

“Moreover, companies hoping to win the race for data talent will need to approach hiring more holistically, removing the stigma and eliminating bias,” the company says. “By implementing smarter hiring processes to identify the right high-quality candidates as opposed to blindly hiring based on university, GPA or SAT scores, the process of building data teams can also become more data-driven.”

Use Better Tools

Data scientists are essentially one-man (or one-woman) bands: They possess all the requisite skills for establishing, building, and implementing at-scale data analytics programs. But just as AI software threatens to automate many jobs currently performed by white collar and blue collar workers, the data scientists’ job itself is also being targeted by software automation.

In particular, the rise of automated machine learning, or “AutoML,” tools raises the possibility that companies can reach their data science goals without employing an actual data scientist (or at least not employing as many of the high-priced unicorns as would otherwise be required).

Google offers its Cloud AutoML offering, but there are many on-premise tools available too

Ashley Kramer, the senior vice president of product management for data science and analytics platform provider Alteryx, has witnessed the data science talent shortage firsthand.

“We see in most of the organizations that have been using our technology and other technology tools that this talent skillset gap has been widening,” she tells Datanami. “There’s just not enough data scientists that exist in the world.”

Thanks to features like preconfigured machine learning models, Alteryx’s platform is able to help a data analyst swing above his weight in the data science realm, Kramer says.

“We have introduced things like pre-configured R and Pyhon tools where any analyst can go in and start building out a logistic regression or a decision tree, but they’re not actually writing code,” she says. “That’s where we have really started to up-level their skills set so they can bridge that gap.”

Alteryx isn’t advocating that data scientists are no longer needed. Instead, the company maintains that companies are better off letting citizen data scientists use AutoML tools to build some relatively well-understood predictive applications, like churn analysis, while keeping the unicorns for the truly tough problems.

“It’s very common to see just a few data scientists in the organizations. They’re generally focused on real high value R&D efforts. Where that ends up hurting the business is they have to focus on those higher level R&D tasks, but who’s building base-level models for the business?” she says. “Data scientists are always going to be in high demand, so we’re trying to figure out what’s the best approach to bridge the gap so business doesn’t fall behind the competition because they’re not able to build these models.”

Hire It Out

If widening the breadth of your data science talent search and up-skilling analysts into “lite data scientists” doesn’t do the trick, there’s one other foolproof method for getting the data science expertise you absolutely need: outsource it.

You can also outsource your data science work (metamorworks/Shutterstock)

The big data boom has been a boon for management consulting firms like Deloitte, McKinsey, Accenture, PwC, KPMG, and Booz Allen Hamilton, all of which have devoted large sums to attracting and retaining top data science talent over the past decade. Of course, customers of these firms can be expected to pay handsomely for the privilege of working with one of their top data people, but that’s to be expected.

Increasingly, a number of smaller specialty firms have started plying the data science personnel waters. One of those is Toptal, a New York-based firm that boasts having the top 3% of talent in any given field. The company recently expanded into advanced analytics and data science

“There are not a lot of people in the market with proven experience,” Toptal’s head of data science and AI Pedro Alves Nogueira recently told Datanami. “Most of our talent is either in the US or in Southern or Eastern Europe.”

According to a recent software development report, the pool of talented IT professionals in Ukraine, Poland, Belarus, and Romania is growing quite quickly at the moment. “While the US and Western Europe are facing the shortage of tech experts, Easter Euproean tech talent pool is contiously growing,” says Andrew Pavliv, the CEO and founder of N-IX, a Ukrainian company that provides data science services, among a wide range of other services.

Which approach you use will depend on your particular situation. In some cases, hiring a high-powered data scientist (or “research scientist” as some of them now identify themselves) may be the right way to establish a data science beachhead. In any event, there’s no reason for companies to sit entirely on the sidelines as the big data revolution continues. You just have to find the approach that works for you.

Related Items:

‘Data Scientist’ Title Evolving Into New Thing

Universities Get Creative with Data Science Education

How To Find and Hire Data Scientists