HBOS datacentre failure causes business continuity questions

A power failure to a datacentre on Saturday stopped all Halifax and Bank of Scotland branches from being able to provide cash machine, over-the-counter and online services for over six hours.

The incident, which occurred during heavy storms and lasted for several hours, frustrated customers and left IT analysts questioning business continuity arrangements at the bank.

The stormy weather cut power at 8am to a vital datacentre of the bank, situated in Yorkshire. Power was restored later in the morning using a back-up generator, but cash machines and other services did not begin working until after 2.30pm, and online banking remained down until 6.30pm.

In the interim, the only transactions customers could make were in branches, as staff wrote down the details on paper.

An HBOS spokesperson apologised to customers, and said the bank had a “well-rehearsed business continuity plan”. It was “extremely rare” for this to happen, the spokesperson said, adding that the engineers were on-site “immediately”.

Customers expressed their frustration on online forums and news websites. “Anything longer than two hours outage is failure of business continuity and disaster recovery policy,” wrote Andy, commenting on the Sky News website.

“Regardless of contingency, perhaps having mirrored virtualised servers in another location (say London) would be the answer,” added Peter on the same site.

Analysts also expressed surprise the systems were down for so long.

“I’m absolutely astonished that a major high street bank could succumb to a power failure like that,” said Trevor LaFleche, senior analyst at IDC Financial Insights. “There seem to have been no proper plans in place for a failover to alternative IT systems or to a new power system.”

He added: “If they were letting things like that go, how sensibly can that be done? You would expect an outage to be for no more than five or 10 minutes.”

Many banks failed to place enough emphasis on testing their disaster recovery, he said. “There are always unexpected scenarios, but something as basic as a power or network problem, you can be totally prepared for that.”

In such a situation, banks needed immediate failovers to other mirrored IT systems, or to alternative temporary power supplies, he said.

“This is a bit of egg on their face for HBOS, and it does raise questions around their business continuity,” said Chris Skinner, chief executive at financial think-tank Balatro. “The arrangements should have been better and you’d expect to switch to another datacentre in real time.”

But many high street banks still only had automatic failover systems in place for their “core banking systems, such as transactional systems”, and not for ATMs and branch banking, he said.

Nevertheless he warned: “During this crisis, with such a loss of confidence in banks, any systems outage just doesn’t help their reputation.”

In June, a disc array problem at Barclays left thousands of customers unable to access their bank accounts online or withdraw money from cash machines for a three hour period.