Continuing from http://status.netnorth.net/2015/05/06/us-based-connectivity-issue/ We just started to receive monitoring reports showing the same level of packet loss as received earlier this morning. We have removed the Level3 connectivity from our network immediately forcing re-routing of traffic. We will contact Level3 for more information.
Level3 based network issues are often very difficult to diagnose. This is due to Level3 operating their BGP network via MPLS (Multi-Protocol Label Switching), a wonderful and useful technology (but difficult to diagnose when your network spans the entire world!) Take this snippet of a traceroute for example: 2 ge-6-18.car1.Manchester1.Level3.net… Continue reading
We have received reports this morning of network alerts from 3rd party providers indicating trouble reaching the Netnorth network. After further investigation, we narrowed the issue to part of the Level3 network (traceroutes included below) located in the US – we raised this with Level3 who confirmed they currently have… Continue reading
At 1:34pm we received alerts regarding some destinations being unreachable. Further investigation showed that our fibre link to Level3 was down unexpectedly. This caused a BGP routing event. Our configuration started the BGP reconvergance process as soon as the router saw the fibre link drop and outbound traffic was re-routed… Continue reading
We experienced a short fibre outage on one of our fibre circuits between Bolton and Manchester overnight. This started at 12:05am and concluded at 12:52am. No maintenance was scheduled for this period so we are awaiting a RFO (reason for outage) from VirginMedia for the circuit. Due to our… Continue reading
We have received a maintenance notification from IXmanchester(LINX) stating that they will be performing switch maintenance on their network. 18/Mar/2015 midnight – 5am (UK local time) This maintenance will affect our IXmanchester peering connectivity during the window. We do not anticipate any disruption from the above maintenance as the exchange… Continue reading
Following some recent reports of very low packet loss on some routes via Cogent, we performed extensive testing of our core infrastructure and located a potential issue with one leg of our metropolitan core ring. (Our core ring traverses four buildings, utilising a different supplier for each leg of the… Continue reading
This evening there have been a few periods of instability with one of our IP Transit providers (Level3). These occurred around 5pm, 7pm and 10:30pm. Level3 have confirmed this is due to a faulty line card which is currently being replaced. These brief periods of instability occur while our routers… Continue reading
Here’s a few thoughts on the recent bash vulnerabilities: A lot of media reports have said only people calling CGI scripts with bash are at risk, but it extends much further than that due to the way the OS works. When spawning a new process (e.g. to start a CGI,… Continue reading
At 8:37pm, switch 23 went offline due to a power failure. Engineers attended site and diagnosed the cause of the power failure and re-established power at 10:28pm and will continue to monitor the power for the next 24 hours. Any customers connected to switch 23 will have seen a network… Continue reading