Image Courtesy: analyticsindiamag.com
We live in a digitally advanced world, where technology is taking massive strides every day and shaping a new future. In a world which now runs on cyberspace and is fully driven by connectedness, we have been producing major chunks of data every moment. While some of this data that proliferates is intentional and is used to run the devices and sustain operations, there is also a chuck, which is produced as a byproduct, goes unnoticed and unutilized, and yet holds the key to transforming the business world. This byproduct data is called dark data. A recent IBM study concluded that about 80% of all data that is produced is dark and unstructured in nature. A systematic innovative approach to the exhaustion of this data by analyzing it might hold an answer to a lot of our technological challenges in the modern world.
Ways to leverage dark data
While systematic and structured data can be used in spreadsheets, tables, and software because it is easy for computers to process and analyze, dark data lurks somewhere unstructured, hidden from our attention spectrums. Dark data is that critical and crucial goldmine which we haven’t yet found a way to utilize properly and make our lives even more automated and easier. This data just sits there idle because its collection has become inexpensive due to cheap cloud storage facilities.
A transformation comes when enough efforts are put into developing methodologies to make dark data meaningful, especially in areas like deep learning, artificial intelligence, cognitive computing, and machine learning technologies. Currently, data from emails, servers, sensors, surveillance videos, semi-structured texts and log files are majorly stored in repositories without yet having a practical usage in our lives.
It would be an accomplishment to have its practical applications through artificial intelligence, wherein machine learning software becomes even smarter. A digital engine can be developed that can handle this semi-structured data from many sources, handle context-sensitive elements to correlate events and process it in real-time. This enriches individual points of that data with additional metrics to make it a meaningful and useful piece of information for big organizations.
Applications of dark data for business
For any business to excel, the Return on Investment (ROI) should be viable. The same is the case with developing capabilities to exhaust the hidden storage repositories of dark data. With immense potential in terms of opening avenues for new revenue generation, there is less risk involved and surfacing dark data for business utilization opens real market opportunities.
For instance, Application Programming Interface (API) and Extensible Markup Language (XML) integration being done on the travel portals have huge in-transit traffic data that can be utilized and interpreted to predict user behaviours and in turn forecast travel preferences and likings of these customers. This will help unlock new and vibrant business avenues. Travel businesses can use these insights to make the customer experience more purposeful.
Further, a massive application can be found in the Internet of Things (IoT) and motion intelligence. A report states that IoT devices may add 269 times more data than what is currently available for use. Here, data mining can be adopted to uncover motion intelligence and location-based data for analytics purposes. For example, mobile phone locations are now determining the relevance of information that is made available to users, with each online search result narrowed down on the basis of the user’s current location and proximity.
Another application can be found in industrial data being converted to operational insights, wherein a lot of hidden data produced by machine sensors can be leveraged in the heavy equipment industry to have real-time equipment fitness information. Running hours and functional load data from these machines can be utilized to anticipate breakdowns and order repairs in time to avert downtime losses. If such heavy machine data through telemetry is picked and utilized, it will enable fleet owners to intervene and take cost-effective decisions on equipment maintenance needs beforehand.
An outlook: Future of dark data usage
The future is data-driven where even the slightest of data points can give a bigger perspective to organizations to improve and grow business, which was earlier out of their spectrum. As the volume and chunk of data now being produced are immense and continue to grow, companies are looking forward to innovative and new ways to have a market edge. Employing the capabilities of data scientists to retrieve this dark data from the most unanticipated sources available is the key to competing in the market. The challenge is now not only to simply leverage this data but to develop meaningful applications that can help your business reach new heights.
(The author is Dr. Anil Kaul, CEO of Absolutdata (an Infogain company) and Chief AI Officer at Infogain and Harshit Parikh, Vice President, Global Practice lead at Infogain and the views expressed in this article are their own)
