Since 11:55pm UK local time, we’ve had unstable IPv6 connectivity via Level3 in Manchester (approx. 60% packet loss).
This has been confirmed from several Level3 router locations (Manchester, London, Amsterdam) and appears to be quite widespread.
As a result, we have temporarily removed our IPv6 connectivity via Level3 pending an update from them regarding this issue.
Note: this does not affect IPv4 traffic, nor IPv6 destinations reachable via our alternate connectivity providers.
Level3 are now investigating our report.
It appears that the unstable connectivity was limited to hosts located with NTT (Nippon Telegraph & Telephone) – as NTT are a large connectivity provider worldwide, this caused a lot of alerts.
We are awaiting an update from Level3 regarding this.
Although Level3 claim that this is resolved (after maintenance on their network), we are still seeing the same issue.
We have provided Level3 with additional traceroutes and await their new response.
We have been escalating this issue with Level3 throughout the day, and their Core IP team are looking into it.
We now see routing loops and missing routing paths which don’t correspond with the advertised paths sent to us.
We continue to push traffic where possible over our alternate connectivity providers.
Update from Level3:
“03/23/2016 15:18:11 GMT – The IP NOC reports that IP services are affected across multiple markets in Europe. Customers are experiencing unreachable destinations. The root cause is being investigated at this time. Tier III is being engaged to assist with investigations.”
The IPv6 instability looks like it may be resolved. We’ve had stable paths for the last hour.
We have received an RFO from Level3, as follows:
“Reason for Outage (RFO) Summary: Network topology changes associated with a scheduled maintenance in the Madrid market resulted in IPv6 (Internet Protocol version 6) routing issues. The scheduled maintenance was performed to move the Wide Area Network (WAN) links on a Madrid, Spain router from one Madrid device to another. Once the maintenance was completed, customer reports of routing issues were received. Level 3 IP engineers made changes to the IPv6 configuration updates on the Madrid router to correct the routing problems. Once the configurations were updated, services were confirmed restored.”