Nutanix Protection Domains Migration With Vmware Vds Fails

Nutanix has this great feature of protection domains and remote sites, which makes it possible to create snapshots of virtual machines and copy these to an other site. This also gives the great possibility to do a failover of VM’s to another site without having to use VMware SRM. Unfortunately in at least version 3.5.2.1 there is a problem when making use of a virtual distributed switch which results in an unavailable VM after a migration. This problems occurs cause every cluster has his own datastore. Nutanix takes care of copying the VMs between datastores. When using a virtual distributed switch, the virtual distributed switch information is also stored on the datastore in a folder called .dvs.

dvs

This data is not taken by the replication process of Nutanix, however I tried copying this data myself but it didn’t give me a working solution.

vDS Nutanix

This results in VMs which are disconnected from the network and you are even unable to enable the connection from the VM settings. The only way to fix this manually is by changing te network or the network port on the virtual distributed switch. Since Nutanix has an extremely powerfull API and VMware delivers also great automation tools with PowerCLI, it was pretty easy to create a script, which automates the proces of a migration and restoring the networking connections. The script is based on the VMware script attached to the KB article below. Related to: VMware KB: 2013639 Make sure PowerCLI is installed, before running this script.

<#

.SYNOPSIS
Migrate Protection Domains which contains VM's that are connected to a VMware Distributed Switch.

.DESCRIPTION
Migrate Protection Domains which contains VM's that are connect to a VMware Distributed Switch.
The script will start a migration from one cluster to another and reconnect the VM to a port on the
appropriate port-group on the distributed switch. This will prevent a VM to become disconnected after
a migration. This problem occurs when migrating between two Nutanix cluster at least in version 3.5.2.1
USE AT OWN RISK! (some parameters are case-sensitive!)

.EXAMPLE
.nutanix_failover_with_dvs.ps1 -vCenter TSTVCR01 -clusterIP NXCLSDTC -pd Test_vDS -remoteSite NXCLSDTD-VDI_BAC
.nutanix_failover_with_dvs.ps1 -vCenter TSTVCR01 -clusterIP NXCLSDTC -pd Test_vDS -onlyFixNic $true

.PARAMETER vCenter
Mandatory parameter which contains the vCenter

.PARAMETER vCenterUser
optional parameter which contains the vCenter user. (default root)

.PARAMETER vCenterPass
optional parameter which contains the vCenter pass. (default vmware)

.PARAMETER clusterIP
mandatory parameter which contains the Nutanix Cluster IP or Hostname, which will fullfill the request.

.PARAMETER clusterPort
optional parameter which contains the Nutanix Cluster Port. (default 9440)

.PARAMETER nutanixUser
optional parameter which contains the Nutanix Cluster username. (default admin)

.PARAMETER nutanixPass
optional parameter which contains the Nutanix Cluster password. (defailt admin)

.PARAMETER pd
mandatory parameter which contains the Protection domain name to migrate.

.PARAMETER remoteSite
mandatory parameter which contains the Remote Site.

.PARAMETER onlyFixNic
optional parameter which doensn't migrate, but only fixes the NIC. (default $false)
#>

# Written by Rob Maas (rob@progob.nl)
# This script is written for a specific use case, so use at own risk.
# Lots of code is strongly based on: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2013639
# 2014-04-11 - First "working?" version

param ([parameter(Mandatory = $true)][string]$vCenter,
                                     [string]$vCenterUser = "root",
                                     [string]$vCenterpass = "vmware",
       [parameter(Mandatory = $true)][string]$clusterIP,
                                     [string]$clusterPort = "9440",
                                     [string]$nutanixUser = "admin",
                                     [string]$nutanixPass = "admin",
       [parameter(Mandatory = $true)][string]$pd,
                                     [string]$remoteSite,
                                     [boolean]$onlyFixNic = $false
       )

#Dirty work-around for self-signed (untrusted) certificates
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;

    public class IDontCarePolicy : ICertificatePolicy {
        public IDontCarePolicy() {}
        public bool CheckValidationResult(
            ServicePoint sPoint, X509Certificate cert,
            WebRequest wRequest, int certProb) {
            return true;
        }
    }
"@

[System.Net.ServicePointManager]::CertificatePolicy = new-object IDontCarePolicy

function ConnectVcenter($vCenter){
   if (connect-viserver -Server $vCenter -User $vCenterUser -Password $vCenterPass -ErrorAction SilentlyContinue){
   #if (connect-viserver -Server $vCenter -ErrorAction SilentlyContinue){
        write-host -ForegroundColor Yellow "Connected to vCenter: $vCenter"
        return $true
   } else {
        write-host -ForegroundColor Red "Unable to connect to vCenter: $vCenter"
        return $false
   }
}

function MigratePD($pd, $remSite){
    $action = "protection_domains/$pd/migrate"
    $uri = "https://$clusterIP`:$clusterPort/PrismGateway/services/rest/v1/$action"
    $body = $remSite
    $header = @{"Authorization" = "Basic "+[System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($nutanixUser+":"+$nutanixPass))}
    #write-host -ForegroundColor Yellow "Executing: $uri"
    write-host -ForegroundColor Yellow "Destination cluster: $body"
    $result = Invoke-RestMethod -Method post -Uri $uri -Headers $header -Body $body
}

#Returns true if an migration is still in progress
function getMigrationState($pd){
    $action = "protection_domains/replications/?ReplicationListFilter=$pd"
    $uri = "https://$clusterIP`:$clusterPort/PrismGateway/services/rest/v1/$action"
    $header = @{"Authorization" = "Basic "+[System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($nutanixUser+":"+$nutanixPass))}
    #write-host -ForegroundColor Yellow "Executing: $uri"
    $result = Invoke-RestMethod -Method get -Uri $uri -Headers $header
    if ($result -ne $null){
        return $true
    } else {
        return $false
    }
}

function getVMs($pd){
    $vms = @()
    $action = "protection_domains/"
    $uri = "https://$clusterIP`:$clusterPort/PrismGateway/services/rest/v1/$action"
    $header = @{"Authorization" = "Basic "+[System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($nutanixUser+":"+$nutanixPass))}
    #write-host -foregroundcolor Yellow "Executing: $uri"
    $tmpResult = Invoke-RestMethod -Method get -Uri $uri -Headers $header
    foreach ($domain in $tmpResult){
        if ($domain.name -eq $pd){
            foreach($vm in $domain.vms){
                #check on (n)
                if ($vm -match '(d+)'){
                    $vm = $vm -replace ' (d+)', ""
                }
                write-host -ForegroundColor Yellow $vm.vmName
                $vms += $vm.vmName
            }
        }
    }
    return $vms
}

Function Get-FreeVDSPort($VDSPG) {
	$nicTypes = "VirtualE1000","VirtualE1000e","VirtualPCNet32","VirtualVmxnet","VirtualVmxnet2","VirtualVmxnet3"
	$ports = @{}

	# Get all the portkeys on the portgroup
	$VDSPG.ExtensionData.PortKeys | Foreach {
		$ports.Add($_,$VDSPG.Name)
	}

	# Remove the portkeys in use  Get-View
	$VDSPG.ExtensionData.Vm | Foreach {
	    $VMView = Get-View $_
		$nic = $VMView.Config.Hardware.Device | where {$nicTypes -contains $_.GetType().Name -and $_.Backing.GetType().Name -match "Distributed"}
	    $nic | where {$_.Backing.Port.PortKey} | Foreach {$ports.Remove($_.Backing.Port.PortKey)}
	}

	# Assign the first free portkey
	if ($ports.Count -eq 0) {
		$null
	} Else {
		return $ports.Keys | Select -First 1
	}
}

#Get the virtual machines
write-host -ForegroundColor Green "Get VMs in Protection Domain (addition '(n)' will be stripped)"
$vms = getVMs -pd $pd
if ($vms -eq $null){
    write-host -ForegroundColor Red "No VMs found!"
    Exit
}

if (!$onlyFixNic){
    #Migrate
    write-host -ForegroundColor Green "Migrate Protection Domain"
    MigratePD -pd $pd -remSite $remoteSite

    Start-Sleep -S 5 #Give it a chance to start

    #Wait till migration is completed
    if (getMigrationState($pd) -eq $true){
        write-host -ForegroundColor Green "Migration started "
        while (getMigrationState($pd)){
            write-host -NoNewline -ForegroundColor Yellow "."
            Start-Sleep -S 15
        }
        write-host
    } else {
        write-host -ForegroundColor Red "Migration failed!"
        Exit
    }

    #Give vCenter some (extra) time
    Start-Sleep 10
}

#Restore network
write-host -ForegroundColor Green "Restore Networkconnections"
if (ConnectVcenter($vCenter)){
    foreach ($machine in $vms){ #Get VMS from PD.
        $vmids = get-vm $machine*   #Get corresponding Vcenter VMS
        if ($vmids.count -gt 1) {
            write-host -ForegroundColor Yellow "Multiple VM's with name: "$vmids
        }
        foreach ($vm in $vmids){
            if ($vm.PowerState -eq "PoweredOn"){
                #Change portID
                write-host -ForegroundColor Green "Fixing: $vm"
                #Check on already failed migrations
                if ($vdspg = Get-VDPortGroup -Name $(Get-NetworkAdapter $vm).NetworkName -ErrorAction SilentlyContinue) {
                    $vdspg = Get-VDPortGroup -Name $(Get-NetworkAdapter $vm).NetworkName
                } else {
                    $vdspg = Get-VDPortgroup | where {$_.key -eq $(Get-NetworkAdapter $vm).NetworkName}
                }
                $portnum = Get-FreeVDSPort -VDSPG $vdspg
                set-networkadapter -NetworkAdapter $vm.NetworkAdapters -PortKey $portnum -Connected $True -DistributedSwitch $vdspg.VirtualSwitch -Confirm:$false | Out-Null
            } else {
                write-host -ForegroundColor Red "Not fixing, due powerstate off: $vm"
            }
        }
    }
}
Rob Maas
Rob Maas
Technical Challanger at ON2IT

If it is broken, fix it! If it ain’t broken, make it better!