At 3:52pm today, one of our Cisco ASR routers experienced a crash within its routing engine. This caused the router to instantly stop routing and any destinations via the router experienced an outage. Unfortunately, this did not just sever connectivity cleanly… it started causing “flapping” (where routes are introduced and… Continue reading
At 11:55pm tonight, our connection to Level3 appeared to disconnect. The symptoms are identical to the previous occasions where the Level3 router rebooted so we assume this to be the case again tonight. The connection is slowly coming back online, which further supports this suspicion. As Level3 appear to be… Continue reading
Yesterday, at around 11:50am, we had some reports of unusual activity on our ring network. We traced this down to elevated levels of CPU usage on our switches. Further investigation pointed to a high level of IGMP traffic being received. We further traced this to our LINX Extreme peering port… Continue reading
Earlier this evening we identified one of our Cisco switch stacks misbehaving causing a constant stream of stack reconverges – this constant reconverge event has been causing layer 2 network instability for traffic flowing via the stack of nine switches. We have just completed a physical inspection of all stack… Continue reading
At 4:35pm we received alerts that one of our hypervisors in Telecity Williams House, Manchester had ceased to process disk activity. No hardware alerts were present, so a forced reboot of the hypervisor was required. The server is now responding normally, and all virtual machines powered back up.
From just before 8am UK local time this morning, we have seen reports of intermittent connectivity issues with both BT and Plusnet. We believe this is due to a power outage in one of the Telehouse datacentres in London. (This is in addition to a reported power outage in Telecity… Continue reading
We have currently lost a resilient path between two of our Manchester datacentre locations. Our building interconnection provider is currently performing maintenance on their equipment, however this outage is unexpected. We have reported it to the vendor for investigation. In the meantime, our TCW site is running at reduced resilience…. Continue reading
At 17:28 (UK Local Time), we noticed a drop in our Level3 connectivity. Level3 remains unavailable to us at this time. We have opened a ticket with them to investigate. Our network has reconverged to use our alternate providers. This would have caused some instability for any routes via Level3… Continue reading
Since 11:55pm UK local time, we’ve had unstable IPv6 connectivity via Level3 in Manchester (approx. 60% packet loss). This has been confirmed from several Level3 router locations (Manchester, London, Amsterdam) and appears to be quite widespread. As a result, we have temporarily removed our IPv6 connectivity via Level3 pending an… Continue reading
The online UPS’s used in our BOL1 datacentre will be taken out of circuit tomorrow for a short maintenance period. The associated Lead Acid batteries are being replaced to ensure extended contingency cover can be provided during any switchover phases between Utility Mains and Local Generation, eg: in the event… Continue reading