Big data, large strides

Santosh DSouzaDirector Systems Engineer NetApp India

At the very outset, let us address two established perceptions around Big Data, which have garnered a lot of media space over the past year or so. The first leads one to believe that Big Data is an emerging trend in the IT industry and that value added deployments of the technology are yet to take off. The second confines the implementation of Big Data technologies to the data analytics space.

Let me try and address these perceptions through an example of a real deployment of the technology. Handling 3.6PB worth of data, generated as a result of tracking its 660 million distinct products, is no easy job in the best of times. To do so in a way that data transforms into actionable information, the CTO of one of the world’s largest consumer cosmetics brands knew that he had to look for unique solutions. His answer lay in creating a master data structure that automatically sorts, organizes, and structures all company data originating anywhere in the world. Once this model was created, development and test engineers used data cloning technologies to create multiple parent datasets and run necessary analyses on them.Finally, by choosing appropriate and innovative data services, business insights were generated in hours instead of weeks. The creation of the master data structure model is just one example of a significant form of implementation of Big Data technologies that happened across the globe in recent times. And analytics was only one part of the leveraging of Big Data technologies.

Making business sense of Big Data, or in practical terms, of any large and disparate sets of data that a particular datacenter or datacenter administrator has no previous experience of effectively managing, has already started making purposeful strides in the world of enterprise IT. Developments in IT infrastructure technologies and improved analytical tools have certainly helped.A lot has been written recently around the gold mine of opportunities that the Big Data conundrum is likely to throw up, mostly from an analytics perspective. Big Data however is no longer anemerging trend as we see it. It is being tackled real time by IT administrators and entails a lot more than analytics alone.There is this whole angle to Big Data around the need for enhanced bandwidth to transmit large data files, the kinds generated due to enhanced collaboration of 3D design processes for instance. Widespread adoption of cloud computing also has a big role to play in increased bandwidth requirements.

There is also the aspect of efficiently storing and managing humongous standalone datasets, which are created daily due to enhanced content consumption via mobility and social media. Creation of master data structure models like the ones by the global cosmetic brand, which organize and sort themselves automatically, almost real time, is a case in point. A multitude of datacenters across the globe are either being redesigned or consolidated to enhance enterprise efficiency and agility using such Big Data technologies. Creation of Enterprise content repositories therefore, also involve widespread deployment of Big Data technologies.

Such implementations of Big Data technologies are already underway. In global capital and commodity markets for example, market intelligence providers are incorporating Big Data technologies to add immense value to what are called ‘market sentiment feeds’. These feeds are helping capital and commodity market players make important trading decisions, thereby impacting the macro economy. Effective implementation of Big Data technologies will be crucial to this sector, given the number of transactions executed in worldwide trade across time zones in a given day. Other sectors where Big Data technologies have penetrated but have escaped public attention till now include the world of sports, for example in trying to ascertain the optimum value of sportspersons; in the world of media and entertainment to manage the lifecycle of large special effects and animation files for global collaboration, or to gain insights intoeffective and innovative programme scheduling.

Finally, national security and law enforcement agencies around the globe haveprobably beenthe biggest consumers of Big Data technologies, given the amounts of data they have to store, manage and analyze from a multitude of sources including surveillance cameras, satellite images, baggage scanners and the like, on an everyday basis. Many of the potential national security threats, which have been diffused globally in the nick of time by federal agencies, are said to be a result of implementations of Big Data technologies.

Big Data has to be viewed as transformational change for a better future, as it offers numerous benefits in terms of effective data storage, predictability and speed of response.  Customers, employees and products - these are the three factors which determines the success of any organization. Big Data helps organizations maintain a competitive advantage by focusing on these growth areas.  By applying analysis of Big Data to pressing business issues, companies are reshaping their operations – and accelerating their business results. By enhancing enterprise agility in terms of efficiently storing large and disparate datasets and generating actionable intelligence in no time, organizations are seeing happier and more productive employees.

India has all but warmed up to the Big Data challenge, and it is important that we stay the course. To do so, it is crucial that IT administrators and decision makers have a broad and deep understanding of the various possibilities that Big Data technologies present and then choose the right implementation to serve the business objectives of their enterprises. As is evident from the examples given above, the world has already started to make large strides in the space. We certainly have opportunities to catch up and perhaps lead in this area.