At approx 2:56pm UK local time today, we saw packet loss across our entire metro network for around 4 minutes. Initial diagnostics showed that one of our core network ring Cisco 3850-series switches had failed (it was stuck in a boot loop with failing self-tests). Ordinarily, this should not have… Continue reading
At 14:56 we had a core switch failure, and a consequential period of reconvergence, lasting approximately 4 minutes. This event will have been service affecting for some cutomers on our network. We are investigating the root cause of the hardware failure, and will publish a fuller RFO on our status… Continue reading
We saw our Vodafone Fibre circuit between BOL8 and LD2 drop at 8:38pm this evening. We have reported the fault to Vodafone to investigate. Our ring automatically re-routed via its alternate paths as expected without any customer outages. More updates will be added via comments as we receive them.
At approx. 1:00pm today, we received a DDoS attack against our primary authorative nameserver (ns3). This DDoS attack was a memcached amplification attack which saturated our external connections and caused packet loss to our Bolton and Manchester networks. By 1:30pm, we had filtered the traffic and all packet loss should… Continue reading
One of our IP transit providers, Cogent Communications, is currently offline. Our connectivity remains unaffected and resillient via alternative IP provuiders and peering. Netnorth customers might have experienced a brief period of instability to some destinations at 12:26 (no more than a couple of seconds during reconvergance), but generally won’t… Continue reading
We currently have a BT Fibre Fault on our metro ethernet ring which Openreach are investigating. As designed, our ring continued to operate at reduced resilience with no loss of service. There are no anticipated service issues as a result of this fault. Once BT have investigated and restored service,… Continue reading
At 6:32pm this evening, we were alerted to a full outage of our network in both Bolton and Manchester. After remote diagnostics failed, an engineer attended one of the sites to perform detailed analysis. Early analysis showed a large amount of traffic operating on our core ring network which looked… Continue reading
At 3:51pm UK local time, we detected a hypervisor reboot in our LD2 Manchester location. All hardware diagnostics confirm that the hardware is performing normally and the cause of the reboot is unknown. The hypervisor is back online, and any affected virtual machines have been restarted.
We have identified a switch stack member failure at 8:15am this morning in our BOL2 location. This is part of our 10G ring and, as such, has caused a break of the resilience in this circuit. Any customer directly connected to this particular switch stack member will see an outage. … Continue reading
At 6:55am this morning, we saw Hypervisor DT-324 crash with a pink-screen-of-death. This has been rebooted after a hardware check and any virtual machines located on this HV are now being booted. As this is the 2nd PSOD on this hypervisor in the last year, we will now take… Continue reading