This evening there have been a few periods of instability with one of our IP Transit providers (Level3). These occurred around 5pm, 7pm and 10:30pm. Level3 have confirmed this is due to a faulty line card which is currently being replaced. These brief periods of instability occur while our routers… Continue reading
At 8:37pm, switch 23 went offline due to a power failure. Engineers attended site and diagnosed the cause of the power failure and re-established power at 10:28pm and will continue to monitor the power for the next 24 hours. Any customers connected to switch 23 will have seen a network… Continue reading
Earlier this evening we experienced some issues with an interlink between Telecity Reynolds House and Telecity Williams House in Manchester. Starting 6:06pm, we started receiving spanning-tree topology errors via the link (provided by Geo) causing the link to flap. This continued randomly until approximately 8:15pm and has been stable since…. Continue reading
Further to our recent ticket, our IP Transit connection to Level3 Communications is offline again. At 1:05am this morning, our physical link to Level3 Communications is reported as down. As this is the second event recently, we have escalated this with the provider. Due to the unclean shutdown, a… Continue reading
We experienced an outage of one of our interlinks between Bolton and Manchester provided by Virgin Media overnight. This fibre-optic link failed to pass traffic between 12:04am and 2:36am. Due to our resilient ring network topology, no outage occurred to customers in Bolton as traffic diverted via an alternate fibre… Continue reading
At 10:53am UK local time, we saw our physical link to Level3 (AS3356) drop for 5 minutes. This caused our routers to immediately terminate the BGP routing session with Level3 and start to reconverge via our alternative paths leading to 2-3 minutes of instability whilst routes updated. The link to… Continue reading
Earlier today, we had a failure of one of our firewalls pairs. This occurred at 2:43pm UK local time. Our firewalls operate in pairs in a failover capacity. Unfortunately, today this failover took much longer than expected. The failover took place within a few seconds as designed, however this triggered… Continue reading
We have received monitoring alerts showing a core problem at one of our IP Transit providers (Level3). While trying to confirm this alert, we can’t reach Level3’s London POP at all so feel that the issue at their end is a lot wider than a local issue. As a result,… Continue reading
We experienced a fibre outage within the BT network this morning at 8:33am. There is a fibre break in one of our core ring circuits between Bolton and Manchester. Our network automatically failed over to its redundant path, and continues to operate normally. Once the fibre break has… Continue reading
At approximately 7:10am this morning, we experience a commerical power failure of the L2 phase of power entering out Bolton 1 facility. This caused our Generator to take over the entire load of the building. Cooling was maintained via ‘free cooling’ (external filtered air travelling through the building via powerful… Continue reading