At approx 1:10pm today, we experienced a network issue.
After investigating, we saw what appeared to be a Layer 2 loop within our network. We have mitigation against these loops, so this was unusual.
After detailed investigation, we discovered that a Cisco 3850 series switch located in Manchester MA2 had crashed but was still forwarding packets.
As the switch software had crashed, it was no longer participating in MST which lead to a Layer 2 loop.
We isolated the switch to recover the rings.
We have been advised to downgrade the version of IOS on the switch from Fuji to Denali. We have done this and partially brought this back into service.
We have plans to migrate from MST to Cisco’s REP (resilient ethernet protocol) technology as our ring topology technology as part of our network upgrades.