Sydney Water error causes shut-downs at Sydney data centre

Macquarie Telecom has admitted a failure of the air conditioning systems at its Sydney-based Intellicentre data centre facility caused a number of customer’s servers to shut down and reset early Saturday morning.

According to Denis Rowe, communications manager at Macquarie Telecom, an emergency response from Sydney Water to an issue with nearby pipes saw an interruption in the water supply to the air conditioning units in the Intellicentre, located located in Pitt Street in Sydney’s CBD.

As a result, temperatures rose in parts of the building causing a number of the servers stored in Macquarie Telecom’s co-location and managed hosting facility to shut down and/or reset.

Rowe said that the downtime for customer servers varied from server to server, ranging from between a number of minutes to a couple of hours.

“We don’t know the exact number of customers affected as many servers automatically reset themselves, but it was a small number,” he said.

“Our account managers get alerts on their phones when this kind of thing happens and they usually outbound [call] their customers straight away. We were pro-actively contacting customers.”

He said customers were also alerted via Macquarie’s online server management tool, which is capable of SMSing customers in the event of an issue with their servers.

Rowe said under Macquarie’s N+1 redundancy scheme, reserve water tanks kicked in following the cutting off of mains water. Once this was depleted, however, the data centre went to an auxiliary source.

“Sydney Water didn’t advise us that this would be happening. Normally we have good communication with them and they advise us of any work, but apparently this was emergency work,” he said. “They have advised us that there is no ongoing problem.”

A spokesperson from Sydney Water said that it had received a call on Friday about a leaking service connection between a mains pipe and a shop located on Pitt Street, Sydney.

“A service crew shut off the water to the business at 9pm Friday, but inadvertently shut off water supplying Macquarie Telecom’s data centre,” the spokesperson said. “Sydney Water was notified of the problem at 8.30am on Saturday [and] a crew returned to the site at 9am Saturday to restore water services to the Telecom building.”

“Sydney Water apologises for any inconvenience caused by the water being shut off.”