Further to our recent ticket, our IP Transit connection to Level3 Communications is offline again. At 1:05am this morning, our physical link to Level3 Communications is reported as down. As this is the second event recently, we have escalated this with the provider. Due to the unclean shutdown, a… Continue reading
We experienced an outage of one of our interlinks between Bolton and Manchester provided by Virgin Media overnight. This fibre-optic link failed to pass traffic between 12:04am and 2:36am. Due to our resilient ring network topology, no outage occurred to customers in Bolton as traffic diverted via an alternate fibre… Continue reading
At 10:53am UK local time, we saw our physical link to Level3 (AS3356) drop for 5 minutes. This caused our routers to immediately terminate the BGP routing session with Level3 and start to reconverge via our alternative paths leading to 2-3 minutes of instability whilst routes updated. The link to… Continue reading
Earlier today, we had a failure of one of our firewalls pairs. This occurred at 2:43pm UK local time. Our firewalls operate in pairs in a failover capacity. Unfortunately, today this failover took much longer than expected. The failover took place within a few seconds as designed, however this triggered… Continue reading
We have received monitoring alerts showing a core problem at one of our IP Transit providers (Level3). While trying to confirm this alert, we can’t reach Level3’s London POP at all so feel that the issue at their end is a lot wider than a local issue. As a result,… Continue reading
We experienced a fibre outage within the BT network this morning at 8:33am. There is a fibre break in one of our core ring circuits between Bolton and Manchester. Our network automatically failed over to its redundant path, and continues to operate normally. Once the fibre break has… Continue reading
At approximately 7:10am this morning, we experience a commerical power failure of the L2 phase of power entering out Bolton 1 facility. This caused our Generator to take over the entire load of the building. Cooling was maintained via ‘free cooling’ (external filtered air travelling through the building via powerful… Continue reading
We were notified of an outage on one of our backup servers (DT-310) this morning at approximately 9:00am UK local time. The server sufferred a hardware RAID controller failure with its HP SmartArray controller. For the sake of data integrity, the filesystem is performing a full FSCK (filesystem check) and… Continue reading
This morning at 8:10am, we started seeing increased latency and packet loss between our network and BT internet. This settled down at 10:34am. Some diagnostics showed the problem to be around the London area, possibly at the interconnect of Global Crossing and BT. BT have raised this as a fault… Continue reading
The server, www10, is restored on temporary hardware.