Harnessing the Information Overload
By Seema Ambastha
Mumbai, Jul 23, 2008

Data Explosion

There is about a 30-40 per cent growth in information every year. Trends imply the information is exploding from kilobytes to petabytes with the help of newer technologies. As the amount of available data increases, the challenge to manage that information gets more difficult, and this can lead to information overload or fatigue.

Take for example the telecom service industry in India. In 2005 - 2006, on an average, 5.5 million cellular lines were added every month, taking the mobile subscriber base to 157 million from 91 million in 2004-2005. (Source: Annual survey conducted by Voice & Data).

In another case, a large wireless service provider has a subscriber base of nearly 5 million subscribers. Imagine, on an average, every subscriber makes 20 calls per day. Each call generates a call detail record (CDR), which is used for calculating the bill and subsequent collection process. Each CDR implies a revenue opportunity for the telecom service provider. Around a 100 million raw CDRs generated daily are transformed before reaching the billing system. All this data is stored online for 3 months and subsequently archived.

IT managers therefore, need to develop skills for managing this information overload.

Managing Data Smartly

Managing huge amounts of data can be a tough challenge. It involves securing, managing and archiving the data in such a way that appropriate executives have access to it and use that data for making informed decisions. This increases the management and storage acquisition costs. Apart from achieving operational goals from the data, new government regulations and guidelines are a key driving force for how and why that data is retained.

An analysis will reveal that 75% of the data is rarely accessed, while barely 25% of it is being frequently used. As the age of the data increases, its access frequency diminishes to almost negligible, if any. Thus, data can be described as being active, less active, historical or ready to be archived. The requirement of information by business users at any given time has forced IT managers to also store it online.

The Solution

In a traditional approach, the data is stored in large storage boxes and once the information is used, it is stored in secondary storage media like tapes. With flat file systems, this sounds simple. In a relational database management systems environment, it requires downtime for both, archival and retrieval. Users have to wait for information to be restored. So managing the information is one of the key requirements in modern day IT enabled business.

Information Life Cycle Management (ILM), the new building block in the IT management strategy is one of the solutions for managing information overload. ILM is a combination of data management capabilities of the database server with appropriate storage technology. Oracle Database enhances support for information lifecycle management, enabling organisations to ensure regulatory compliance, optimise storage, streamline business processes and reduce cost.

India landscape

Organisations that manage very large volumes of data require self-managing databases. They need to carry out database upgradation with minimal disruption and have to deal with a large number of security and compliance issues. Financial services, communications and government organisations in India stand to gain greatly from using a database with an ILM platform.

Industry analysts claim that the worldwide relational database management systems market is poised for continued growth through 2010 as data management and integration become strategically important in an increasing number of organizations.

The market will witness healthy growth, reflecting ongoing demand for relational database management systems, especially for online transaction processing, which have traditionally been the drivers of growth in the past. It will also experience increased demand from organisations opting for business intelligence and data warehousing technologies to manage their data effectively.


The author is the director (database sales consulting) of Oracle India