When even DR failed

by Ivor Soans    Aug 05, 2005

Starting on the afternoon of Tuesday, July 26, the city of Mumbai experienced a disaster on a scale it has never witnessed before. Thanks to an instance of extreme rainfall, the city received what is perhaps the most amount of rain any densely populated metropolis has ever received, anywhere in the world.

True to fashion, the government failed spectacularly — first by not issuing any warning even when they had data that suggested that the rain was of the severest kind, and then, by doing a disappearing act. As a satirical cartoonist put it, “There was no politician’s name on the list of the dead, but they were all there on the list of those missing.”

But then, that is hardly surprising in India. It’s the exception, rather than the norm, when a government does its work in the wake of a disaster, leave alone doing a good job — one reason why the Tamil Nadu government was praised so highly for a good job in handling the aftermath of the tsunami eight months ago.

But what surprised me is that the much-vaunted private sector failed too. The private telecom companies were in poorer shape as compared to the government-owned operator, but while the government owned operator admitted to large-scale damage, the private guys pretended as if nothing had happened. At our own office we experienced a complete failure of landlines as well as Net access from Tata Indicom, even as I read a quote from a senior executive of the company claiming there had been no disruption.

And then there were some banks. They surprised me the most — it was quite impossible for banks to replenish their ATMs in Mumbai, but surprisingly the banking system across the country got affected too, because most banks are headquartered in Mumbai. So, my friend in Bangalore couldn’t offer customers at his restaurant the credit card facility because the system went on the blink and his bank said it occurred because of systems failing in Bombay; ATMs failed in Delhi, and the sob stories just went on.

What happened to the disaster recovery centers that were built for just such a day? Weren’t they supposed to take over and ensure that though Mumbai was marooned the rest of the country wouldn’t catch a cold?

In many ways, key IT systems failed in Mumbai last week and enterprises need to undertake urgent remedial measures to ensure this never happens again. In many cases, it is also about people policies. Your DR center could be based in Chennai, but if your System Administrator has the passwords and authentication mechanism to get things transferred to the Chennai center, but is sitting in a mammoth traffic jam on a Mumbai road, with water swirling all around and with no communication links working, then your DR center isn’t going to do any disaster recovery that day.

What happened in Mumbai on July 26 was a wake-up call that Indian enterprise IT can ill afford to ignore.