We experienced another power outage of phase L3 in our BOL2 datacentre this morning at 3:09am.
Electricity Northwest are currently on-site tracing an underground cable fault.
It is hoped it will be within a section that can be isolated to restore power to the building.
The datacentre is currently running from generated power, and no outages are expected.
Staff are onsite monitoring the generators to ensure continued service.
ENWL are still tracing the underground cable fault.
They were unable to isolate enough of the circuit to restore power temporarily.
ENWL are currently liasing with their excavation team to begin lifting the cables at the probably fault locations.
They may bring temporary generators on site whilst mains power is unavailable (this outage affects multiple customers of ENWL)
We continue to monitor our on-site generation, and no outages are expected.
We have sufficient on-site fuel for many days of power generation, and a contract for fuel delivery.
Commercial power was restored to site at 6:33am, but the ENWL engineer is still onsite. We will remain in contact with them until they leave the site.
There is likely to be further power outages once they start excavating.
Our generators are back on automatic standby and have sufficient fuel reserves for several days of backup power generation.
no services have been affected by this commercial mains outage.
We have received an update from ENWL – they are analysing the remote monitoring of the electricity circuit to diagnose the suspected point of failure below ground. This may provide them with multiple potential failure points.
An excavation crew is en-route to dig where necessary.
The on-site ENWL engineer suspects the fault to lie in a road nearby and says that it is the most likely point that they will dig.
The circuit is complicated by the length of the circuit from the substation to all its users.
The engineer has warned that the circuit is likely to trip again. Our automated systems will take care of this should it happen and no disruption to customer services is expected.
ENWL will dig and sever the cable (so that the fault hopefully lies beyond the live circuit) and cap the end.
Whilst a possibility, ENWL do not believe the fault is on the circuit to our building directly.
ENWL are currently excavating in a street nearby. Commercial power remains available, and backup generation is primed.
ENWL have completed their excavating and have severed the cable and capped the ends.
Both ENWL and Netnorth will monitor the power over the next few days/weeks and hopefully this will resolve the recent issues.