Level3 Routing Issue

At 17:28 (UK Local Time), we noticed a drop in our Level3 connectivity.

Level3 remains unavailable to us at this time.  We have opened a ticket with them to investigate.

Our network has reconverged to use our alternate providers.  This would have caused some instability for any routes via Level3 (but not via our other paths) during the convergence.

Bookmark the permalink.

5 Responses to Level3 Routing Issue

  1. Level3 have confirmed their IP NOC team are investigating.

  2. We just saw our physical link re-establish – this is looking increasingly like a Level3 router reboot.

  3. Our Level3 IPv6 connectivity has been restored (this has minimal impact to our network due to the small routing table).

    We will keep IPv4 administatively shutdown for now until Level3 confirm the reason for outage to us.

  4. Level3 have issued the following update:

    “The Level 3 IP Network Operations Center (NOC) responded to alarms indicating a possible unresponsive device in Manchester, United Kingdom that is impacting IP traffic. The IP NOC is investigating at this time and has engaged Level 3 Field Services for an onsite investigation.”

    As a result, we will keep our Level3 IPv4 connectvity administratively down until they report back with further information to prevent potential additional instability.

  5. We have received the following update from Level3:

    “The Level 3 IP Network Operations Center (NOC) responded to alarms indicating a possible unresponsive device in Manchester, United Kingdom that impacted IP traffic. The IP NOC investigated and was able to remotely access and reboot the device, corrected a memory issue, thus restoring service. Service remains stable at this time and continues to be monitored for stability. This service impact has concluded; however, there may be additional analysis and discovery that occurs as the incident is reviewed by NOC management.”

    Whilst they consider the service stable, we will not re-establish our Level3 connectivity for several hours to ensure that we’re confident that it’s stable too.

Leave a Reply