At 7:04am this morning, we had a dark fibre failure between two of our Manchester datacentres. The cause of this is currently being investigated by the fibre provider. Unfortunately, our backup fibre path failed to take over leading to one of our Manchester datacentres becoming isolated. This was caused by… Continue reading
At approx 4:55pm UK local time, we experienced a loss of our LINX (London) connectivity. This is provided by a “Connexions” partner, and we suspect an issues lies somewhere within their network. A brief period of instability would have occurred whilst BGP reconverged automatically to our failover paths. No further… Continue reading
At approx 6:45am this morning, our monitoring triggered reports of instability for connectivity via our LINX connection (London Internet Exchange) in London. As this link was not saturated in any way, we took the decision to shutdown the link to cause all traffic to re-route via alternative paths. This connection… Continue reading
At 1:43pm, our monitoring identified a link failure on our metro ethernet ring between Bolton and Manchester. As designed, our ring reconverged via backup links and no outage was experienced – however the ring now has reduced resilience. We have raised a fault with Virgin Media to investigate the cause… Continue reading
At approx. 8:15pm this evening, our alerting saw a flap of our protected fibre link between Manchester and London (provided as an MPLS pseudowire) which affected connectivity between LDeX2 and LDeX1 datacentres. This link is used solely for our connectivity to LINX LON1 and LINX LON2 peering networks, and caused… Continue reading
At approx 10:35pm this evening, one of our core Cisco routers unexpectedly rebooted. This was due to an IOS-XE crash. The router automatically restarted, and BGP sessions will re-establish. Some network instability would have occurred during this restart due to the unclean nature of the reboot. We will investigate… Continue reading
We currently have a 10Gbit/s Openreach fibre circuit between BOL23 and MA1 offline. Openreach are investigating. Our ring automatically rerouted traffic to prevent an outage to customers. No outages are anticipated, but the ring is operating at reduced resilience in the interim.
At 11:15am, Bolton suffered a large power cut which caused a momentary loss of grid power to our DC. This was handled by our UPS systems as expected and was brief enough to not require backup generation. Electricity northwest (ENWL) are en-route to the area to investigate so a further… Continue reading
At approx 1:10pm today, we experienced a network issue. After investigating, we saw what appeared to be a Layer 2 loop within our network. We have mitigation against these loops, so this was unusual. After detailed investigation, we discovered that a Cisco 3850 series switch located in Manchester MA2 had… Continue reading
At 10:07pm this evening, our monitoring detected the loss of our Metro Ethernet Ring links from one our suppliers. Our ring automatically reconverged as designed. At 10:13pm this evening, we saw these links reconnect. No planned maintenance is known, so we have raised a query with the supplier and… Continue reading