We experienced another power outage of phase L3 in our BOL2 datacentre this morning at 3:09am. Electricity Northwest are currently on-site tracing an underground cable fault. It is hoped it will be within a section that can be isolated to restore power to the building. The datacentre is currently running… Continue reading
Further to yesterday’s outage, we were alerted to another identical outage this evening at 11:26pm. Once again, our automated failover systems migrated the site to locally generated power. We are awaiting an update from ENWL, our regional energy supplier, with regards to a time to repair.
We experienced a power outage of phase L3 on the commercial mains input at our BOL2 site at approx. 5:45pm this afternoon. The automated failover procedure occurred as expected and the site transferred over to locally generated power. The entire site became powered by generator for the duration of the… Continue reading
Continuing from http://status.netnorth.net/2015/05/06/us-based-connectivity-issue/ We just started to receive monitoring reports showing the same level of packet loss as received earlier this morning. We have removed the Level3 connectivity from our network immediately forcing re-routing of traffic. We will contact Level3 for more information.
We have received reports this morning of network alerts from 3rd party providers indicating trouble reaching the Netnorth network. After further investigation, we narrowed the issue to part of the Level3 network (traceroutes included below) located in the US – we raised this with Level3 who confirmed they currently have… Continue reading
At 1:34pm we received alerts regarding some destinations being unreachable. Further investigation showed that our fibre link to Level3 was down unexpectedly. This caused a BGP routing event. Our configuration started the BGP reconvergance process as soon as the router saw the fibre link drop and outbound traffic was re-routed… Continue reading
We experienced a short fibre outage on one of our fibre circuits between Bolton and Manchester overnight. This started at 12:05am and concluded at 12:52am. No maintenance was scheduled for this period so we are awaiting a RFO (reason for outage) from VirginMedia for the circuit. Due to our… Continue reading
At 11:59 the commercial power was restored to all 3 phases. This was detected by our switchgear, and after an observed period of stability, the generator was put on standby and normal service was reinstated. We do not expect any further interruption, but will continue to monitor the service. Netnorth… Continue reading
Circa 10:46 one of the three commercial phases to the Queensbrook estate failed somewhere upstream of the on-site distribution. This supplies our BOL1 U8 datacentre and our administrative offices, no other Netnorth datacentres or infrastructure have been affected. The partial supply failure was identified by our switchgear, and the standby… Continue reading
Following some recent reports of very low packet loss on some routes via Cogent, we performed extensive testing of our core infrastructure and located a potential issue with one leg of our metropolitan core ring. (Our core ring traverses four buildings, utilising a different supplier for each leg of the… Continue reading