News

posted on 28 July 2008 20:12

The Malaysian Bursa Stock Exchange had to suspend trading when a disk drive on an HP NonStop system failed, according to reports. The Bursa's CIO has since resigned.

To expect a single drive failure to cause an HP NonStop system to crash beggars belief.

The fault ocurred on July 4th and a day's trading was lost, costing an estimated RM450,000 in clearing fees. It was the worst failure since June, 2000, when trading was halted for half a day. A commentator said the IT system should be fail-safe; it wasn't, and the backup system also failed.

A timeline shows a hard disk failure at 5.30am. It was replaced with another disk at 5.35am. By 6pm the replacement disk was also facing problems and other disks failed followed by a system CPU. The system was restarted at 6.30am but brokers experienced connection failures and half of the Exchange's brokers couldn't connect by 8am. Trading was suspended at 8.30am.

By 1pm it was evident that the backup process was taking longer than expected, said to be due in part at least to integrity measures being enacted. A primary site restart was unsuccessful at 1.20pm and the afternoon trading was lost. Normal trading resumed on the next day.

How can a single hard disk failure stop a NonStop system? How can a backup system make matters worse unless it was a very poor backup system?

It is quite obvious that the Bursa didn't check out its backup[ strategy and look for vulnerabilities in it using software such as Continuity's RecoverGuard. Apparently Bursa ran four disaster recovery tests a year. Much good it did them.

Yew Kim Keong, the Bursa's chief information officer has since resigned, with effect from July 31st. He has taken personal responsibility for the failure. Bursa CEO Datuk Yusli Mohamed Yusoff said the exchange hopes to launch a new equities trading system in the next two months, using software from Atos Euronext Market Solutions.

Phew, no need to really get into why the current system crashed then. Be great to understand the local HP tech support view of the crash.

^{[Martin Edwards, news writer.]}