3rd Party Maintenance (12/Feb/2017)

We have received a maintenance notification from Cogent Communications regarding one of IP Transit connections to the global internet.

This maintenance is due on 12th February 2017 between midnight and 7am, with an expected outage time of 45-60 minutes.

No outage to Netnorth customers is expected as we have multiple IP Transit connections to the global internet.  Our routers should gracefully re-route any affected traffic paths to our other connectivity providers.

Emergency Router Maintenance (28/Jan/2017)

Following on from the router crash experienced yesterday, we have performed an emergency upgrade of the Cisco IOS-XE software on all of our internet-facing routers.

This update should resolve the issue that was triggered yesterday.

No outage to customers should have been experienced today, but there may have been brief periods of slightly increased latency during route changes.

Cisco Router Crash – 27/Jan/2017

At 3:52pm today, one of our Cisco ASR routers experienced a crash within its routing engine.

This caused the router to instantly stop routing and any destinations via the router experienced an outage.

Unfortunately, this did not just sever connectivity cleanly… it started causing “flapping” (where routes are introduced and removed over and over again causing instability).  Once this flapping was identified, we severed all network connectivity to the affected router.

After a few minutes, BGP failover took over and traffic re-routed via alternative paths as it is designed to do.  This is how a normal crash would be handled.

The router crashed in such a way that it had to be physically power cycled to regain control afterwards.  We then brought its routing online in a slow and controlled fashion to prevent any further disruption to the network.

 

After some research, it appears that we hit CSCus82903 which is a known Cisco Bug in our edition of routing software.

This was triggered when attempting to bring online our new IP connectivity provider, GTT, this afternoon – a normally routine procedure with no impact to customer traffic.

 

Our routers are currently stable and operating normally, however we need to perform some emergency maintenance to upgrade the software of the routers to a patched version provided by Cisco.

This should be able to occur without causing any additional outages, although the network routing should be considered “at risk” during the actual software upgrade.

In the meantime, our GTT connection has been kept offline to prevent the issue reappearing.  We will re-establish the connection once the software upgrades are complete.

Network Connectivity

We’re seeing packet loss on the LINX Extreme network this morning.
We’ve also got alerts coming in for our London location too, this is routed entirely separately to our Bolton/Manchester locations, so it looks like it is affecting the entire LINX LAN.

We’ve shunted most traffic over to the LINX Juniper network so this should die down now. We’ve not had any official word from LINX yet, however we will continue to monitor and migrate traffic paths where possible.

Netnorth Support

Storms / Power – Bolton

There is currently a storm over the Bolton area (with very nice fork lightning!) which has caused a brief power outage this evening around 7pm UK local time.

Our UPS units continued to operate during the outage with no loss of power to the datacentres, and our generators started to take over the load if required.

This was not required as the mains was restored within a few seconds.

Our generators returned to their idle state after a few minutes once mains power stability was confirmed.

 

Should there be any further outages, the generators will re-start automatically.  The process to transfer power to generator in a mains fail situation is fully automated.

Advisory – ‘TeamViewer’ vigilance

‘TeamViewer’ – A Collaborative Meeting and Remote-Control tool

Some customers who choose to deploy TeamViewer may be affected by recently reported events. This may actually be an indirect vulnerability, in that users have been compromised elsewhere within their daily social activities, allowing ‘authenticated access to their TeamViewer infrastructure’ to be used as an attack vector.

Customers who use TeamViewer to access and control their hosted servers at Netnorth are advised to maintain a vigilant password regime, and to observe the TeamViewer press releases.

For more information see:

If you have any further questions, please email support@netnorth.co.uk

Regards
Netnorth Support

Planned Maintenance – BOL1

The online UPS’s used in our BOL1 datacentre will be taken out of circuit tomorrow for a short maintenance period.

The associated Lead Acid batteries are being replaced to ensure extended contingency cover can be provided during any switchover phases between Utility Mains and Local Generation, eg: in the event of any  Utility failure. This planned maintenance should not affect any hosted services, but will mean that some services will be subject to minimal ‘elevated risk’ for the duration of the maintenance.

Further notices will be posted during the maintenance window.

Regards
Netnorth Limited

Advisory – ‘DROWN’ (Cross-protocol attack on TLS using SSLv2) vulnerability

‘DROWN’ – Decrypting RSA with Obsolete and Weakened eNcryption

Various products may be affected by this vulnerability, given the common vulnerability and exposures designation of CVE-2016-0800.

A cross-protocol attack was discovered that could lead to decryption of TLS sessions by using a server supporting SSLv2 and EXPORT cipher suites. Traffic between clients and non-vulnerable servers can be decrypted provided another server supporting
SSLv2 and EXPORT ciphers (even with a different protocol such as SMTP, IMAP or POP) shares the RSA keys of the non-vulnerable server.

Users can avoid this issue by disabling the SSLv2 protocol in all their SSL/TLS servers, if they’ve not done so already. Disabling all SSLv2 ciphers is also sufficient, provided the patches for CVE-2015-3197 (fixed in OpenSSL 1.0.1r and 1.0.2f) have been deployed.  Servers that have not disabled the SSLv2 protocol, and are not patched for CVE-2015-3197 are vulnerable to DROWN even if all SSLv2 ciphers are nominally disabled, because malicious clients can force the use of SSLv2 with EXPORT ciphers.

For more information see:

Or search for
‘ CVE-2016-0800 ‘

If you have any further questions, please email support@netnorth.co.uk

Regards
Netnorth Support

Advisory – ‘Skeleton’ glibc Vulnerability

‘Skeleton’ Buffer Overflow Vulnerability

Various products may be affected by this vulnerability, given the common vulnerability and exposures designation of CVE-2015_7547.

Multiple vulnerabilities in the ‘libresolv’ library of the ‘GNU C Library’ (aka ‘glibc’), could allow a remote attacker to remotely execute code. This typically affects systems based on Linux derived platforms, or platforms operating a ‘Linux compatability’ mode, which in turn resolve DNS queries using ‘glibc’.

We are bringing this vulnerability to the attention of our customers in light of recent press releases. Note that Netnorth resolvers limit the UDP packet length to mitigate this issue, but customers who may have deployed affected software / hardware AND who choose to resolve DNS themselves or externally, will need to ensure that relevant patches are applied to avoid exposure.

For more information see:

Or search for
‘ CVE-2015_7547 ‘

If you have any further questions, please email support@netnorth.co.uk

Regards
Netnorth Support

Advisory – Cisco ASA Vulnerability

Cisco ASA Software IKEv1 and IKEv2 Buffer Overflow Vulnerability

Various Cisco ASA products may be affected by this vulnerability, given the common vulnerability and exposures designation of CVE-2016-1287.

A vulnerability in the Internet Key Exchange (IKE) version 1 (v1) and IKE version 2 (v2) code of Cisco ASA Software could allow an unauthenticated, remote attacker to cause a reload of the affected system or to remotely execute code. This vulnerability affects systems configured in routed firewall mode only and in single or multiple context mode. This vulnerability can be triggered by IPv4 and IPv6 traffic.

Cisco has released software updates that address this vulnerability. We will embark on a series of recommended upgrade paths to the software on relevant managed Cisco products that we host. This will involve minimal downtime during the necessary reboot of the ASA devices concerned.

For more information see:

https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20160210-asa-ike

Or search for
‘ CVE-2016-1287 ‘

If you have any further questions, please email support@netnorth.co.uk

Regards
Netnorth Support