Skill Shortage Impeding Big Data Adoption

by CXOtoday News Desk    Feb 27, 2015

big data

Big data and Hadoop are popular buzzwords in the industry today. However, most companies are uncertain about how and when to implement these technologies, claimed a joint study conducted by SnapLogic and TechValidate, which added skill shortage is a major cause impeding its growth and creating clouds of uncertainly.

The study, based on a survey of 100 IT leaders from large enterprises highlighted that IT leaders are keen on Big data’s ability to power sharper analytics and other modern applications, but struggle with limited skills and resources. In other words, despite the keenness, there is still a significant shortage of skilled professionals who can truly be called Data Scientists who can evaluate business needs and impact, write the algorithms and program platforms such as Hadoop.

In one of its recent reports, McKinsey & Co. Too has placed the shortfall of big data experts at anywhere from 140,000 to 190,000 by the year 2018.

According to the survey, big data integration technologies continue to be among the top few priority areas of IT investment. Top drivers for investing in Big data solutions include customer analytics (52 percent), operational analytics (40 percent), data-driven products and services/Internet of Things (38 percent).  However, over 43 percent cite limited skills and resources as the biggest hindrance of big data technology, followed by compliance and security (41 percent), and data fragmentation (34 percent).

According to the study, the majority of the respondent said they didn’t know what Hadoop tool was in use or for planned use, some plan to use MapReduce, Hive, Pig, Spark and others in the coming days. Again, the majority indicated they didn’t know of Hadoop distribution in use or for planned use but others are using (or plan to use) Apache Hadoop, Cloudera, MapR and Hortonworks.

“Our survey shows there’s a good bit of indecision right now when it comes to Big data plans and technologies. At the same time, the results show strong interest for using Big data to achieve business goals, using integration data sets to quickly understand a customer’s changing needs,” said Darren Cunningham, vice president of marketing at SnapLogic.

 ”By 2015, 4.4 million IT jobs globally will be created to support big data, says Peter Sondergaard, senior vice president at Gartner and global head of Research. And therefore companies need to gear up on big data.

Enterprise tech commentator and writer, Rick Delgado notes lack of academic capability as one of the key reasons for skill shortage in big data. He believed it is important for academia and the industry to identify skills that are needed now and in future to meet the challenges and that include computing and analytical skills. “Some companies are geared more toward a specific expertise in Big Data, particularly being able to understand it, collect it, and preserve it,” he says.