There is currently a storm over the Bolton area (with very nice fork lightning!) which has caused a brief power outage this evening around 7pm UK local time.
Our UPS units continued to operate during the outage with no loss of power to the datacentres, and our generators started to take over the load if required.
This was not required as the mains was restored within a few seconds.
Our generators returned to their idle state after a few minutes once mains power stability was confirmed.
Should there be any further outages, the generators will re-start automatically. The process to transfer power to generator in a mains fail situation is fully automated.
Yesterday, at around 11:50am, we had some reports of unusual activity on our ring network.
We traced this down to elevated levels of CPU usage on our switches. Further investigation pointed to a high level of IGMP traffic being received.
We further traced this to our LINX Extreme peering port in London and shut the port down. This resolved the issue instantly.
We contacted our Connexions provider, who provide our LINX ports. They did investigation to ensure the traffic was not originating internally and then passed the query onto LINX.
LINX confirmed this morning that they had a member port injecting IGMP traffic to the peering LAN yesterday and that this has now been resolved.
We have re-enabled our LINX Extreme port and confirmed the issue no longer exists.
We will re-enable our peering connections via this LAN shortly.