20/09/2013 Some VM’s unrepsonsive – Resolved
Date: 20/09/13Time first reported: 03:08Time resolved: 08:17Details: We received an alarm from the storage cluster reporting network connectivity issues to one node. Connections have failed over as expected but one of the XEN hosts failed to establish a new connection. This resulted in some VM’s on this host becoming unresponsive. This problem wasn’t immediately obvious
13/09/13 UPS Maitenance at Risk – resolved
Date: 13/09/2013At risk maintenance window: 16/09/2013 21:00 – 17/09/2013 05:00Time first reported: NATime resolved: 17/09/2013 We have now had confirmation that this work has been completed without any issuesDetails: We have gone the extra mile and added our own backup power supply to further reduce the risk of the required maintenance on the Datacentres UPS.
28/08/13 – Failed disk in storage node caused some VMs to become unresponsive – resolved
Date: 28/08/13Time first reported: 10:58Time resolved: 12:00Details: A failed disk was reported in one of our storage nodes. This caused a momentary pause to the storage and all connections failed over as expected. The interruption to the storage appears to have been less than 30 seconds. In this case the host running some customer
09/08/2013 Host down Temperature sensor failure – Resolved
Date: 09/08/2013Time first reported: 03:09Time resolved: 03:40Details: Host went offline due to an incorrect temperature sensor warning which has caused some VM’s to powered offImpact: Some VM’s down and not respondingUpdate: This was an isolated issue on one of the physical hosts. VM’s were automatically recovered to other hosts. The Host has now been taken out of production
06/08/2013 XEN host down – Resovled
Date: 02/08/2013Time first reported: 09:18Time resolved: 10:05Details: Host went offline which has caused some VM’s to powered offImpact: Some VM’s down and not respondingUpdate: This was an isolated issue on one of the XEN hosts. The Host has now been taken out of production and VM’s have been powered back on.
02/08/2013 Virtual Machines disconencted -resolved
Date: 02/08/2013Time first reported: 00:05Time resolved: 00:50Details: Host went offline which has caused some VM’s to become disconnectedImpact: Some VM’s down and not respondingUpdate: We found that the host had two failed disks, causing the host to become unresponsive. Also HA did not automatically recover VM’s. VM’s have now been manually started on other hosts. We check
28/06/2013 Storage upgrade at risk notification -Resolved
Date: 28/06/13Time first reported: 09:15Time resolved: 22:32Details: We have experienced a few outages related to storage connectivity during the last couple of weeks. In some situations this has required unplanned restarts of virtual machines. We have been working with HP and Emulex support teams to pinpoint the cause of the issue. The issue only seems
27/06/13 – Loss of Internet Connection – Resolved
Date: 27/06/13 Time first reported: 14:59 Time resolved: 15:02 Details: We have experienced a brief loss of internet connection with our ISP. This has had no affect on the actual Virtual Machines but their connection to the internet would have been interrupted for a few minutes. The connection was restored after just a few minutes.
26/06/13 – High disk latency on some VMs reported
Date: 26/06/13 Time first reported: 12:00 Details: We are currently experiencing some disk latency issues with some VMs on our cluster. We are addressing the issue and will provide further updates when we have them. Please note that this is only affecting a handful of VMs.