At 11:55pm tonight, our connection to Level3 appeared to disconnect. The symptoms are identical to the previous occasions where the Level3 router rebooted so we assume this to be the case again tonight. The connection is slowly coming back online, which further supports this suspicion. As Level3 appear to be… Continue reading
We’re seeing packet loss on the LINX Extreme network this morning. We’ve also got alerts coming in for our London location too, this is routed entirely separately to our Bolton/Manchester locations, so it looks like it is affecting the entire LINX LAN. We’ve shunted most traffic over to the LINX… Continue reading
Yesterday, at around 11:50am, we had some reports of unusual activity on our ring network. We traced this down to elevated levels of CPU usage on our switches. Further investigation pointed to a high level of IGMP traffic being received. We further traced this to our LINX Extreme peering port… Continue reading
Earlier this evening we identified one of our Cisco switch stacks misbehaving causing a constant stream of stack reconverges – this constant reconverge event has been causing layer 2 network instability for traffic flowing via the stack of nine switches. We have just completed a physical inspection of all stack… Continue reading
From just before 8am UK local time this morning, we have seen reports of intermittent connectivity issues with both BT and Plusnet. We believe this is due to a power outage in one of the Telehouse datacentres in London. (This is in addition to a reported power outage in Telecity… Continue reading
We have currently lost a resilient path between two of our Manchester datacentre locations. Our building interconnection provider is currently performing maintenance on their equipment, however this outage is unexpected. We have reported it to the vendor for investigation. In the meantime, our TCW site is running at reduced resilience…. Continue reading
At 17:28 (UK Local Time), we noticed a drop in our Level3 connectivity. Level3 remains unavailable to us at this time. We have opened a ticket with them to investigate. Our network has reconverged to use our alternate providers. This would have caused some instability for any routes via Level3… Continue reading
Since 11:55pm UK local time, we’ve had unstable IPv6 connectivity via Level3 in Manchester (approx. 60% packet loss). This has been confirmed from several Level3 router locations (Manchester, London, Amsterdam) and appears to be quite widespread. As a result, we have temporarily removed our IPv6 connectivity via Level3 pending an… Continue reading
We have received notification from BT that one of our metro interlink circuits needs an outage on 8th February 2016 starting at 11pm until 6am UK local time as the fibre optic cabling needs to be re-routed for National Rail engineering works. This circuit forms part of our dual metro… Continue reading
This morning we experienced an increase latency and packet loss event at our bolton and manchester locations due to external factors beyond our control. This occured between 03:02am and 03:06am, and again between 03:09am and 03:13am UK local time. During the above windows, latency was increased above normal levels, and… Continue reading