3rd Party Maintenance (12/Feb/2017)

We have received a maintenance notification from Cogent Communications regarding one of IP Transit connections to the global internet.

This maintenance is due on 12th February 2017 between midnight and 7am, with an expected outage time of 45-60 minutes.

No outage to Netnorth customers is expected as we have multiple IP Transit connections to the global internet.  Our routers should gracefully re-route any affected traffic paths to our other connectivity providers.

Emergency Router Maintenance (28/Jan/2017)

Following on from the router crash experienced yesterday, we have performed an emergency upgrade of the Cisco IOS-XE software on all of our internet-facing routers.

This update should resolve the issue that was triggered yesterday.

No outage to customers should have been experienced today, but there may have been brief periods of slightly increased latency during route changes.

Cisco Router Crash – 27/Jan/2017

At 3:52pm today, one of our Cisco ASR routers experienced a crash within its routing engine.

This caused the router to instantly stop routing and any destinations via the router experienced an outage.

Unfortunately, this did not just sever connectivity cleanly… it started causing “flapping” (where routes are introduced and removed over and over again causing instability).  Once this flapping was identified, we severed all network connectivity to the affected router.

After a few minutes, BGP failover took over and traffic re-routed via alternative paths as it is designed to do.  This is how a normal crash would be handled.

The router crashed in such a way that it had to be physically power cycled to regain control afterwards.  We then brought its routing online in a slow and controlled fashion to prevent any further disruption to the network.

 

After some research, it appears that we hit CSCus82903 which is a known Cisco Bug in our edition of routing software.

This was triggered when attempting to bring online our new IP connectivity provider, GTT, this afternoon – a normally routine procedure with no impact to customer traffic.

 

Our routers are currently stable and operating normally, however we need to perform some emergency maintenance to upgrade the software of the routers to a patched version provided by Cisco.

This should be able to occur without causing any additional outages, although the network routing should be considered “at risk” during the actual software upgrade.

In the meantime, our GTT connection has been kept offline to prevent the issue reappearing.  We will re-establish the connection once the software upgrades are complete.

3rd Party Maintenance (23/Feb/2017)

We have received a maintenance notification from Virgin Media regarding one of our metro fibre circuits between Bolton and Manchester.

This maintenance is due on 23rd February 2017 between midnight and 7am, with an expected outage time of 20 minutes.

No outage to Netnorth customers is expected as we have multiple metro fibre links via diverse paths via diverse fibre providers.  Our network will automatically re-route any traffic via our other fibres during the outage.

Generator Tests – BOL

After a minor change in operating procedure, we briefly neglected to post the results of our weekly generator tests. For completeness, here’s a list of the intervening tests…

28-08-2016
BOL1  10:12 - 10:38  Off load  Passed
BOL2  10:56 - 11:13  Off load  Passed

05-07-2016
BOL1  10:24 - 10:42  On load   Passed
BOL2  10:53 - 11:16  On load   Passed

12-07-2016
BOL1  10:20 - 10:49  Off load  Passed
BOL2  11:02 - 11:22  Off load  Passed

19-07-2016
BOL1  10:15 - 10:25  Off load  Passed
BOL2  10:55 - 11:14  Off load  Passed

28-07-2016
BOL1  10:08 - 10:32  Off load  Passed
BOL2  10:41 - 11:03  Off load  Passed

The generator was started, then ran off-load / on-load for the durations given above, before detecting mains and shutting down in the expected timeframe.
All measured values were within their normal ranges.