At 6:32pm this evening, we were alerted to a full outage of our network in both Bolton and Manchester. After remote diagnostics failed, an engineer attended one of the sites to perform detailed analysis. Early analysis showed a large amount of traffic operating on our core ring network which looked… Continue reading
At 3:51pm UK local time, we detected a hypervisor reboot in our LD2 Manchester location. All hardware diagnostics confirm that the hardware is performing normally and the cause of the reboot is unknown. The hypervisor is back online, and any affected virtual machines have been restarted.
We have identified a switch stack member failure at 8:15am this morning in our BOL2 location. This is part of our 10G ring and, as such, has caused a break of the resilience in this circuit. Any customer directly connected to this particular switch stack member will see an outage. … Continue reading
At 6:55am this morning, we saw Hypervisor DT-324 crash with a pink-screen-of-death. This has been rebooted after a hardware check and any virtual machines located on this HV are now being booted. As this is the 2nd PSOD on this hypervisor in the last year, we will now take… Continue reading
At 3:52pm today, one of our Cisco ASR routers experienced a crash within its routing engine. This caused the router to instantly stop routing and any destinations via the router experienced an outage. Unfortunately, this did not just sever connectivity cleanly… it started causing “flapping” (where routes are introduced and… Continue reading
At 11:55pm tonight, our connection to Level3 appeared to disconnect. The symptoms are identical to the previous occasions where the Level3 router rebooted so we assume this to be the case again tonight. The connection is slowly coming back online, which further supports this suspicion. As Level3 appear to be… Continue reading
Yesterday, at around 11:50am, we had some reports of unusual activity on our ring network. We traced this down to elevated levels of CPU usage on our switches. Further investigation pointed to a high level of IGMP traffic being received. We further traced this to our LINX Extreme peering port… Continue reading
Earlier this evening we identified one of our Cisco switch stacks misbehaving causing a constant stream of stack reconverges – this constant reconverge event has been causing layer 2 network instability for traffic flowing via the stack of nine switches. We have just completed a physical inspection of all stack… Continue reading
At 4:35pm we received alerts that one of our hypervisors in Telecity Williams House, Manchester had ceased to process disk activity. No hardware alerts were present, so a forced reboot of the hypervisor was required. The server is now responding normally, and all virtual machines powered back up.
From just before 8am UK local time this morning, we have seen reports of intermittent connectivity issues with both BT and Plusnet. We believe this is due to a power outage in one of the Telehouse datacentres in London. (This is in addition to a reported power outage in Telecity… Continue reading