This evening there have been a few periods of instability with one of our IP Transit providers (Level3).
These occurred around 5pm, 7pm and 10:30pm.
Level3 have confirmed this is due to a faulty line card which is currently being replaced.
These brief periods of instability occur while our routers re-calculate alternate paths to destinations via our other providers. During this time, all ISPs around the world also recalculate paths to us via our alternate routes. These can last from a few seconds to a few minutes.
We balance our connectivity across multiple providers, so these instabilities affect certain destinations but not all destinations.
Update: Level3 have confirmed the line card is currently being replaced, which was the cause of the instability at 10:30pm.
Our Level3 connectivity has re-established and loaded a full routing table, however we are awaiting official confirmation from Level3 that the line card has been changed and no errors are being seen.
The Level3 circuit is down once again.
Level3 confirm that the line card replacement is still ongoing.
Due to the disruption this is causing, we have shutdown our connectivity via Level3 until we receive notification of the replacement line card.
Connectivity will automatically reroute via our alternate providers automatically.
Level3 have reported that the line card has now been replaced, and all errors cleared. We will wait a short while before reinstating the circuit.
Our Level3 connectivity has been re-established after 1 hour with no further error reports. We will monitor overnight.