We have received reports of some unusual packet loss to certain destinations since 2:10am this morning. We have conducted testing within our network to confirm that the issue lies outside of our network infrastructure. As a result, we have raised the issue with our global connectivity providers to investigate further.
Our monitoring systems alerted us to IPv4 connectivity issues at 6:32pm this evening. This was traced to our connection with Level3, Inc. It appears to have stopped passing IPv4 traffic due to a routing loop internally within Level3’s network. We have shutdown our connectivity with them and raised a trouble… Continue reading
We experienced another power outage of phase L3 in our BOL2 datacentre this morning at 3:09am. Electricity Northwest are currently on-site tracing an underground cable fault. It is hoped it will be within a section that can be isolated to restore power to the building. The datacentre is currently running… Continue reading
Further to yesterday’s outage, we were alerted to another identical outage this evening at 11:26pm. Once again, our automated failover systems migrated the site to locally generated power. We are awaiting an update from ENWL, our regional energy supplier, with regards to a time to repair.
We experienced a power outage of phase L3 on the commercial mains input at our BOL2 site at approx. 5:45pm this afternoon. The automated failover procedure occurred as expected and the site transferred over to locally generated power. The entire site became powered by generator for the duration of the… Continue reading
Continuing from http://status.netnorth.net/2015/05/06/us-based-connectivity-issue/ We just started to receive monitoring reports showing the same level of packet loss as received earlier this morning. We have removed the Level3 connectivity from our network immediately forcing re-routing of traffic. We will contact Level3 for more information.
We have received reports this morning of network alerts from 3rd party providers indicating trouble reaching the Netnorth network. After further investigation, we narrowed the issue to part of the Level3 network (traceroutes included below) located in the US – we raised this with Level3 who confirmed they currently have… Continue reading
At 1:34pm we received alerts regarding some destinations being unreachable. Further investigation showed that our fibre link to Level3 was down unexpectedly. This caused a BGP routing event. Our configuration started the BGP reconvergance process as soon as the router saw the fibre link drop and outbound traffic was re-routed… Continue reading
We experienced a short fibre outage on one of our fibre circuits between Bolton and Manchester overnight. This started at 12:05am and concluded at 12:52am. No maintenance was scheduled for this period so we are awaiting a RFO (reason for outage) from VirginMedia for the circuit. Due to our… Continue reading
At 11:59 the commercial power was restored to all 3 phases. This was detected by our switchgear, and after an observed period of stability, the generator was put on standby and normal service was reinstated. We do not expect any further interruption, but will continue to monitor the service. Netnorth… Continue reading