Network Platforms - unexpected VMNode reboots – Incident details

All systems operational

unexpected VMNode reboots

Resolved
Major outage
Started 27 days agoLasted 1 day

Affected

Virtual Hosting

Major outage from 3:12 AM to 5:15 AM, Partial outage from 3:12 AM to 5:15 AM, Under maintenance from 3:12 AM to 5:15 AM, Operational from 3:12 AM to 5:15 AM

CT1 - iSCSI SAN

Under maintenance from 3:12 AM to 5:15 AM

CT1 - VMWare Nodes

Operational from 3:12 AM to 5:15 AM

CT1 - vSAN

Operational from 3:12 AM to 5:15 AM

JB1 - iSCSI SAN

Major outage from 3:12 AM to 5:15 AM

JB1 - VM Nodes

Partial outage from 3:12 AM to 5:15 AM

Updates
  • Resolved
    Resolved

    Good Day

    The scheduled maintenance on the virtual hosting infrastructure has been completed and all system are stable at the moment.

    We continue monitoring the resource usage on the cluster and our automated resource balancer will move workloads between nodes to ensure that there is no oversubscription of resources on a single node.

    Regards

    Network Platforms

  • Identified
    Identified

    Good day

    We are investigating an issue where two of our VMNodes restarted unexpectedly after the scheduled maintenance work had been completed. This affected nodes 146(twice) and node 147. The nodes rebooted within 30minutes of each other resulting in HA restarting the VMs on other available nodes.

    Preliminary investigation suggest that an oversubscription of memory is the cause and our engineers are busy manually rebalancing workloads on the cluster to ensure that no single node is oversubscribed.

    Regards

    Network Platforms