We experienced a power outage of phase L3 on the commercial mains input at our BOL2 site at approx. 5:45pm this afternoon. The automated failover procedure occurred as expected and the site transferred over to locally generated power. The entire site became powered by generator for the duration of the… Continue reading
Continuing from http://status.netnorth.net/2015/05/06/us-based-connectivity-issue/ We just started to receive monitoring reports showing the same level of packet loss as received earlier this morning. We have removed the Level3 connectivity from our network immediately forcing re-routing of traffic. We will contact Level3 for more information.
We have received reports this morning of network alerts from 3rd party providers indicating trouble reaching the Netnorth network. After further investigation, we narrowed the issue to part of the Level3 network (traceroutes included below) located in the US – we raised this with Level3 who confirmed they currently have… Continue reading
At 1:34pm we received alerts regarding some destinations being unreachable. Further investigation showed that our fibre link to Level3 was down unexpectedly. This caused a BGP routing event. Our configuration started the BGP reconvergance process as soon as the router saw the fibre link drop and outbound traffic was re-routed… Continue reading
We experienced a short fibre outage on one of our fibre circuits between Bolton and Manchester overnight. This started at 12:05am and concluded at 12:52am. No maintenance was scheduled for this period so we are awaiting a RFO (reason for outage) from VirginMedia for the circuit. Due to our… Continue reading
At 11:59 the commercial power was restored to all 3 phases. This was detected by our switchgear, and after an observed period of stability, the generator was put on standby and normal service was reinstated. We do not expect any further interruption, but will continue to monitor the service. Netnorth… Continue reading
Circa 10:46 one of the three commercial phases to the Queensbrook estate failed somewhere upstream of the on-site distribution. This supplies our BOL1 U8 datacentre and our administrative offices, no other Netnorth datacentres or infrastructure have been affected. The partial supply failure was identified by our switchgear, and the standby… Continue reading
Following some recent reports of very low packet loss on some routes via Cogent, we performed extensive testing of our core infrastructure and located a potential issue with one leg of our metropolitan core ring. (Our core ring traverses four buildings, utilising a different supplier for each leg of the… Continue reading
This evening there have been a few periods of instability with one of our IP Transit providers (Level3). These occurred around 5pm, 7pm and 10:30pm. Level3 have confirmed this is due to a faulty line card which is currently being replaced. These brief periods of instability occur while our routers… Continue reading
Earlier this evening we experienced some issues with an interlink between Telecity Reynolds House and Telecity Williams House in Manchester. Starting 6:06pm, we started receiving spanning-tree topology errors via the link (provided by Geo) causing the link to flap. This continued randomly until approximately 8:15pm and has been stable since…. Continue reading