We have received the official reason for outage from Level3 Communications for the brief 5 minute outage on our Level3 IP connectivity yesterday (12th September 2012) shortly after midnight.
The outage was caused by routine maintenance by Level3 who had failed to notify us in advance. This has been resolved for future to ensure that we receive advanced warning of maintenance works.
Level3 Communications provide our primary IP connectivity path. During the 5 minute outage, our secondary path automatically took over.
We are currently upgrading the capacity on our secondary IP transit connection from Cogent Communications. This circuit is being increased to 1000mbps.
The new fibre optic interlink has been installed, the configuration will be complete by the end of September 2012.
This link provides both IPv4 and IPv6 connectivity to our network.
One of our web servers (www10) is currently offline due to a hardware failure. The event started around noon.
We are currently restoring the data to a new piece of hardware, ETA just over 1 hour.
Our monitoring systems detected a circuit failure between Bolton 1 and Manchester 2 locations, supplied by Virgin Media Business.
A ticket has been lodged with the supplier for them to investigate a potential fibre fault.
All traffic was automatically re-routed via our alternate path with Cable&Wireless in the meantime.
We’ve started to increase our AVS (anti-virus and anti-spam system) scanners recently.
This is our latest scanner node, a HP BL465c quad quadcore Blade with 32GB of ram. While this might seem rather excessive for a bit of email scanning, the technology behind our scanners is very CPU and memory intensive. Our scanners scan each email in seconds, matched tens of thousands of rules against the textual content of the email. We also use various optical character recognition methods to decipher image attachments.