Deduplication Technology to Attract Telecom

by Sonal Desai    Jun 09, 2009

N K Subramaniyam, executive director(operations and technology) of Saksoft in an interaction with Sonal Desai discusses on the current market update on ‘de-duplication’ technology, and also clearifies some mis-understanding facts on the same

 Please give an overview of the current data de-duplication market in India?

Deduplication processes are widely used in India in the Banking and Insurance domains and now in Telecom. The key areas are Risk management, Customer Service and Sales and Marketing. Risk management is done by ensuring the credit worthiness of the customer at the application origination stage itself. From a customer service perspective, the Deduplication process saves time for manually identifying and fixing duplicate records and also saves money by eliminating wasted direct mailers.


Is the term data de-duplication rightly understood by the CIO?

There are two widely understood definitions of data Deduplication and its business application. Data Deduplication from a data storage perspective is a method of reducing storage needs by eliminating redundant data.

Deduplication from the perspective of Customer Centricity is a set of algorithms that identify distinct records across different sources and build record linkages. This term and its importance in making better business decisions is well understood by the CIO.

Can you give me some examples on how data de-duplication processes have helped organisations in India?

In India, accuracy and consistency of customer data across multiple systems is an area of concern. In the absence of a single version of the truth for customer data, organizations may take additional exposure on existing customers. Deduplication process enables better Risk Management by controlling credit quality at the application origination stage as well as managing additional exposure to potentially risky customers. Sophisticated algorithms result in record linkages across a wide variety of data sources, both internal and external, and create a Unified Customer View. The Unified Customer View can be used to prevent or minimize exposure to potentially risky customers and also for product cross-selling and up-selling. This is also important to adhere to compliance and regulatory requirements and other key areas like Anti- Money Laundering monitoring.

What kind of investment does the technology (Dedupe) require? What can be the ROI (Return on Investment) & TCO (Total cost of ownership) benefits? Are those calculative?

Most Product Processors, or Transaction Processing systems have some rudimentary Deduplication capability, but this is an internal functionality of the system and does not work across systems to create cross-system record linkages.  Hence the Deduplication solution has to be implemented across the organization to effectively manage risk.

The Dedupe system can either be implemented in the network as a locally hosted application, or can be offered as a SaaS (Software as a Service) model. This will depend on the customer s budget for Capital expenditure. The Total Cost of Ownership changes as per the solution implementation model.

Return on Investment for Deduplication application can be measured in terms of improved credit quality, reduced losses due to providing credit to potentially risky customers and a Unified Customer View that may be used for product cross selling and up-selling also.

How mature is the Indian market for such a concept?

The Indian market needs sophisticated Deduplication processes, and unlike in the US, UK and some other countries, there is no unique customer identifier like a Social Security Number or National Identity Number. Passport nos, Driving License numbers, PAN Cards etc may be treated as surrogates for a unique identifier, but their usage is still limited and quality of data is suspect.  Further, addresses in the UK can be checked against the Royal Mail Post Code File for record linkages, in India usage of Census data for Deduplication and data cleansing is still not prevalent.

Traditionally, customer- centric risk management and pricing programs has been the forte of the Banking industry, but now the telecom and insurance industry is also realizing the importance of maintaining a Unified Customer View.

What are some of the mis-understanding facts related to the concept data-duplication ?

There is a key misinterpretation of data Deduplication depending on the business context of usage .Data Deduplication from a data storage perspective (often called "intelligent compression" or "single-instance storage") is a method of reducing storage needs by eliminating redundant data. Only one unique instance of the data is actually retained on storage media, such as disk or tape.

From a customer data management and customer service perspective, Deduplication is the process of identifying duplicate data whereby a data set can be matched against a reference data set to identify matching data using multiple algorithms and variable matching thresholds.

It is therefore important to understand the differences between the two terminologies as both refer to two different concepts altogether.


Which are the industries that have adopted data-duplication processes? How does this technology integrate with the other business processes in an organisation?

Sophisticated deduplication processes to identify duplicate data across multiple source systems is critical for high customer data centric segments like Banking, Financial services, Insurance, Telecom, Airlines and Credit bureaus. Lack of Deduplication leads to costly business decisions based on inaccurate data, and further may require expensive retrospective Deduplication.


What will be the new trends shaping this technology in India. Which will be the new industry and focus?

 Deduplication of customer records is a key requirement in any organization that deals with customer data- ex: Banking, Financial services, Insurance, Telecom, Travel, Airlines etc. Traditionally the Banking industry has been at the forefront of building and maintaining a Unified Customer View, but now other industries are following.

In India, one of the important drivers will be the Credit Bureaus which will maintain the customers record across relationships with multiple financial service providers and hence must be able to deduplicate across disparate data sources.

With new technologies evolving each day, where do you see the data de-duplication technology ten years down the line? What are the changes you envision?

With evolving technologies, Deduplication processes will become more robust and will provide greater accuracy of matches. As mentioned earlier, credit bureaus that will build consolidated customer records will be key drivers to better credit quality for each financial services provider.

However when unique identifiers like a Social Security no are issued and mandated in India for all transactions, then it is expected that there will be reduced need for  an external enterprise-wide Deduplication solution.