A data storage hardware failure was to blame for day-long Tokyo Stock Exchange outage on Thursday October 1.
In a press conference, TSE executives “squarely accepted responsibility for the incident, rather than trying to deflect blame onto the system vendor Fujitsu Ltd,” Bloomberg reports. “All the responsibility lies with us as the market operator, TSE CEO Koichiro Miyahara said. “Fujitsu is merely a vendor that supplies the equipment.”
Fujitsu spokesman Takeo Tanaka said: “We apologise for any inconvenience caused to the concerned parties because of a failure in the hardware we delivered.” Fujitsu is working with the TSE to prevent a recurrence of the problem.
This is a refreshing example of companies taking it on the chin.
Failover failure
The Tokyo Stock Exchange (TSE) runs on a Fujitsu-developed Arrowhead hardware and software system. On Thursday, a data storage component, called the Number 1 Shared Disk device, detected a memory error.
in the event of primary database failure, the database software was supposed to initiate an automated switchover to the secondary database. However, the automated failover process to the Number 2 Shared Disk device did not happen.
TSE IT staff manually forced a change over to the Number 2 device. But they then faced a total system reboot to start trading, with ongoing trading orders left hanging and incomplete.
That was unacceptable and the exchange, the third largest in the world, had to shut down while a proper recovery took place.
The Shared Disk device has a central role, storing management data used by 400 Fujitsu Primergy RX2540 M4 system servers in the Arrowhead trading system. It also handles commands and ID/password combinations for terminals that monitor trades.
Arrowhead uses Fujitsu’s PRIMEFLEX for HA Database, a converged infrastructure system, with software running on integrated Fujitsu hardware. This includes PCIe-connected SSDs and an ETERNUS storage array. There are primary and secondary databases synchronised through mirroring technology. Fujitsu discontinued the PRIMEFLEX for HA Database in March 2017.
