At approx 4:24pm this evening we saw a failure of a SAN shelf in one of our BOL8 SANs used for provisioning virtual machines. This shelf failure caused a loss of array #7 on the SAN which affected a handful of virtual machines.
Upon investigation, an air conditioning unit failed in the room which is catered for by a redundant unit. However, this caused the fan modules in the SAN shelf to increase and triggered a failure of the fan units which caused the shelf to shutdown. The SAN quarantined the array to protect the data on it.
We replaced the fan units in the shelf, and applied some extra airflow while bringing the shelf back online.
The array is now back online and any affected virtual machines have been restarted. The array will take up-to 24 hours to reconstruct and check the data on the array in the background during which time disk I/O may be slower than usual for that array.
We continue to monitor the temperatures on the hottest day of the year so far. The ambient temperatures at BOL8 and BOL23 are the highest we have ever recorded for the last 20 years.