At 1:34pm we received alerts regarding some destinations being unreachable.
Further investigation showed that our fibre link to Level3 was down unexpectedly. This caused a BGP routing event.
Our configuration started the BGP reconvergance process as soon as the router saw the fibre link drop and outbound traffic was re-routed via other paths.
Other routers on the internet would take around 3-4 minutes to re-learn the alternate paths for inbound. This is out of our control.
This only affected paths via Level3 and not via our other connectivity partners and as such would appear as a partial outage.
We have contacted Level3 and raised a ticket to investigate. Our network is fully operational via alternate connectivity.
Level3 have provided an update stating that their router that we connect to in Manchester unexpectedly reloaded.
They are investigating the root cause of this.
While they investigate, we have kept our Level3 connectivity offline to prevent any further reconverging events.
Once we receive an all-clear from Level3, we will bring this circuit back into operation.
Our network is fully operational, and is designed to operate with connectivity failure events such as this one.
We have received the following update from Level3:
“The IP NOC reports that the router has remained stable after it was reloaded. All associated alarms are clear and customer services have been restored.”
Although Level3 have cleared the circuit for operation, we will not bring this circuit back online yet. We prefer an extended observation period before re-use.
After extended observation, there have been no further issues with the Level3 connection. We have re-implemented the connectivity into the network.
We will continue to monitor into the evening.