Getting Big Data's Big Opportunity Right


The irony of big data today is that despite the tremendous hype in the market from the vendor and analyst community through successful use cases and survey results, many businesses still do not know what to expect from it and IT does not have a clear understanding of how to derive benefits from it.

Despite this irony, there is no doubt that enterprises today have a huge amount of existing data that contains valuable insights which can increase revenue, reduce costs and improve customer loyalty.

The challenge is to determine where to look for the right data sets, how to extract meaningful insights from the glut of information that exists and how to interpret this in the right context.

While traditionally business has looked at smaller, business driven structured data systems such as CRM, ERP, and SCM to achieve these results, a large amount of information today resides within unstructured data generated from social media, weblogs from human activity and also images, video and  audio from machine activity such as satellites and sensors.

While traditional business data sets are in the range of one to tens of TBs, these newer data sets are typically in the range of tens to hundreds of TBs.

These factors mandate a different set of technologies to extract insights at a faster pace than with current technologies. Sometimes, while organizations might have to work with the same data sets, the difference is the need to crunch the same data in a much shorter time to improve efficiency and enable faster decision making.

It is because of these differences, that the newer information assets have been categorized as Big Data.

While there is a huge market projection for technologies that allow the extraction of insights from Big Data over the next few years in India as well as globally, the cases described below will help demystify the opportunity.

Recently, a large utility company that provides smart meters to consumers turned to analysis of its smart meter data. Their objective was to analyze variable consumer demand and provide variable pricing based on current and historical demand to optimize consumption. At the same time, they were looking to determine the impact on power infrastructure such as feeders and transformers at different load levels to enable accurate load balancing and avoid outages.

This smart meter data is growing in hundreds of terabytes, and the loads can be analyzed every 30 minutes now. With earlier systems, in comparison, the pricing and loads were analyzed only based on quarterly historical data.

Similarly, a public safety agency uses analysis of video footage available from car accident sites with the objective of identifying patterns occurring in those incidents that could be avoided to improve public safety. This data, at 12PB capacity already, can now readily be used for cross-incident analysis. Until this point, individual post incident analysis could only be performed once a day, without the ability to share across agencies.

Health and life science companies are looking for cures for diseases such as cancer by analyzing genome sequences, clinical trial data as well as inputs available from patients in various medical communities.

Energy production companies use satellite images and seismological data to understand the geological patterns that could lead to the discovery of oil and gas.

Financial organizations are using the audit trails available from weblogs to detect fraudulent transactions.

Supply chains are being made more efficient by a better understanding of which transportation routes to take based on weather forecasts or material availability and prices in a particular region.

There are many more similar use cases that highlight the tremendous potential of big data technologies as potent change makers. The patterns and insights that can be unearthed from existing data sets can be invaluable to businesses. However, while analyzing and extracting these insights is important, one has to be able to apply these in the right context using human intelligence and judgment in order to derive the most accurate results and benefits.