Big data to drive high performance computing

by Sohini Bagchi    Jun 24, 2013

HPCbig data

In recent times, big data has changed the way CIOs and data managers leverage information. However, as big data continues to grow, standard computing CPUs will not be able to process the exponential volume of information. As a result, high performance computing (HPC), often called supercomputing is found to be particularly valuable for organizations with demanding big data requirements. A recent research from analyst firm IDC also substantiates that a growing number of HPC sites are using big data techniques in their work.

HPC meets big data

Until recently, HPC was restricted to high-tech research and academia and was disconnected from the enterprise IT spectrum. However, with the convergence of big data and HPC, the commercial segment can be highly benefited, according to researchers. As Rod Vawdrey, President of the Global Business Group & Corporate Senior Vice President of Fujitsu said in an interview that HPC is coming closer to the mainstream, thanks to the convergence of big data, cloud and even mobile trends.

“I think supercomputing was something that was always thought of as something that was used in research and academia. But, with the advance of technology and the price performance equation, we now can bring it into commercial applications,” says he.

IDC notes cloud computing is also being used for HPC workloads, with the proportion of sites exploiting cloud computing rising from 13.8 percent in 2011 to 23.5 percent in 2013. It found public and private cloud use about equally represented among the 2013 sites.

Experts believe that data-driven businesses with a constant thirst for storing and analyzing large quantities of data will force suppliers to develop big data-friendly solutions. Some leading HPC manufacturers have begun developing cost-efficient, accessible devices that can help companies support big data requirements more effectively. These types of solutions provide a combination of storage capacity and processing performance necessary to maximize the value of information.

Growth opportunities

Among HPC sites, co-processors and accelerators are seen to have increasingly gaining momentum. Those sites using co-processors or accelerators have grown from 28.2 percent in 2011 to 96.9 percent in 2013, which is a big number. “There is a substantially increased penetration of co-processors and accelerators at HPC sites around the world, along with the large proportion of sites that are applying big data technologies and methods to their problems,” says Earl Joseph, IDC’s program vice president.

The most popular among them are Intel Xeon Phi co-processors and NVIDIA GPUs. “HPC has been a growth area for Intel, especially in the past one year. In India currently the technology is driven by the government by the department of Science and Technology, space, defense and a whole range of scientific research. Going forward it will be a huge area of interest for the enterprise,” says Srinivas Tadigadapa, director, Enterprise Solution Sales, Intel South Asia. Tadigadapa sees an increasing closeness or synergy between high performance computing and big data and there is immense opportunity for growth and innovation when used strategically.

Moreover, IDC noted that storage is the fasting-growing technology area at HPC sites and will continue to play a pivotal role in big data for the next few years as businesses become increasingly data-driven. It also forecasts that revenue for HPC servers acquired primarily for big data use will approach $1 billion in 2015. The other observation is that HPC vendors are increasingly targeting commercial markets, while at the same time, commercial vendors, such as Oracle, SAP and SAS, are seeing a lot of opportunities in HPC. Currently, most of them are working on cases that include fraud detection, genomics and personalized medicine. Going forward, researchers see industries such as manufacturing, media, healthcare, financial and automation to increasingly embrace HPC to solve their critical big data challenges.