One of our IP transit providers, Cogent Communications, is currently offline. Our connectivity remains unaffected and resillient via alternative IP provuiders and peering. Netnorth customers might have experienced a brief period of instability to some destinations at 12:26 (no more than a couple of seconds during reconvergance), but generally won’t… Continue reading
We currently have a BT Fibre Fault on our metro ethernet ring which Openreach are investigating. As designed, our ring continued to operate at reduced resilience with no loss of service. There are no anticipated service issues as a result of this fault. Once BT have investigated and restored service,… Continue reading
At 6:32pm this evening, we were alerted to a full outage of our network in both Bolton and Manchester. After remote diagnostics failed, an engineer attended one of the sites to perform detailed analysis. Early analysis showed a large amount of traffic operating on our core ring network which looked… Continue reading
At 3:51pm UK local time, we detected a hypervisor reboot in our LD2 Manchester location. All hardware diagnostics confirm that the hardware is performing normally and the cause of the reboot is unknown. The hypervisor is back online, and any affected virtual machines have been restarted.
We have identified a switch stack member failure at 8:15am this morning in our BOL2 location. This is part of our 10G ring and, as such, has caused a break of the resilience in this circuit. Any customer directly connected to this particular switch stack member will see an outage. … Continue reading
At 6:55am this morning, we saw Hypervisor DT-324 crash with a pink-screen-of-death. This has been rebooted after a hardware check and any virtual machines located on this HV are now being booted. As this is the 2nd PSOD on this hypervisor in the last year, we will now take… Continue reading
At 3:52pm today, one of our Cisco ASR routers experienced a crash within its routing engine. This caused the router to instantly stop routing and any destinations via the router experienced an outage. Unfortunately, this did not just sever connectivity cleanly… it started causing “flapping” (where routes are introduced and… Continue reading
At 11:55pm tonight, our connection to Level3 appeared to disconnect. The symptoms are identical to the previous occasions where the Level3 router rebooted so we assume this to be the case again tonight. The connection is slowly coming back online, which further supports this suspicion. As Level3 appear to be… Continue reading
Yesterday, at around 11:50am, we had some reports of unusual activity on our ring network. We traced this down to elevated levels of CPU usage on our switches. Further investigation pointed to a high level of IGMP traffic being received. We further traced this to our LINX Extreme peering port… Continue reading
Earlier this evening we identified one of our Cisco switch stacks misbehaving causing a constant stream of stack reconverges – this constant reconverge event has been causing layer 2 network instability for traffic flowing via the stack of nine switches. We have just completed a physical inspection of all stack… Continue reading