PowerCLI New-VM GuestOSCustomization fails when script reads an XML file

I have a Powershell (v5.1) script (in 32-bit mode) that I have been using to create new VMs (our vRealize orchestration seems to fail a log). I'm running PowerCLI v5.5.

The script works great, the only problem with it is that a lot of the data is hardcoded in the script. I wrote an XML file to contain all of the variable data that I want to choose from, but when I added code to read the XML file, the VMs would fail to change the IP address during OSGuestCustomization (and apparently then admin account credentials) and thus the new VM was rather useless to me.

It is perplexing why adding this line:

$script:CMDBInfo = [xml](gc c:\src\Enterprise\Enterprise\Systems\Scripts\Powershell-SCMDEV\Opsbrain\TMCMDB.xml)

to the script would cause this line to fail:

$create_vm = New-VM -Template $script:json_data.opb_template -ResourcePool $script:json_data.opb_build_cluster -Name $script:json_data.host_name -Datastore $datastore -DiskStorageFormat Thin -OSCustomizationSpec $customization -Confirm:$false -RunAsync -ErrorAction Stop$create_vm = New-VM -Template $script:json_data.opb_template -ResourcePool $script:json_data.opb_build_cluster -Name $script:json_data.host_name -Datastore $datastore -DiskStorageFormat Thin -OSCustomizationSpec $customization -Confirm:$false -RunAsync -ErrorAction Stop

I finally tried simply commenting out the GC and the New-VM worked.

A brief search turned up https://communities.vmware.com/thread/482488 and I mounted the disk on another server to examine the VMware temporary files and logs.

I see some errors, but they aren't immediately apparent what their causes are.

c:\windows\setupact.log : dispci.dll:  DispCISkipClassInstaller: SetupDiGetSelectedDriver failed with error 0xe0000203.

c:\windows\setuperr.log was empty.

c:\windows\temp\vminst.log : 
2017-09-29 14:06:06| tools-build-3917699| INFO: Failed to fetch component state for '_driver_memctl.5C48FA9C_E001_43CB_B2B6_590CD422177E'.
2017-09-29 14:06:40| inst-build-3917699| E1: FILE: FileDeletionRetry: Non-retriable error encountered (C:\Windows\TEMP\vmware-SYSTEM\00007bf9\windows.iso): The system cannot find the file specified (2)

but it looks like it finished ok:

2017-09-29 14:06:40| inst-build-3917699| I1: Upgrader finished execution, now signalling event.
2017-09-29 14:06:40| inst-build-3917699| I1: Upgrader: returning [0]

I stripped out a lot of comments and 'extraneous' data to get a representative sample of the XML file.

<?xml version="1.0" encoding="utf-8"?>
<CMDB>
    <IPTable>
        <Network Name="DevServerPrivate">
            <Range Subnet="192.168.1.0" RangeLow="192.168.1.21" RangeHigh="192.168.1.150" SubnetMask="255.255.255.0" DefGW="192.168.1.1" DNS="8.8.8.8,8.8.4.4" VMwareNetworkLabel="devserver" />
        </Network>
    </IPTable>
</CMDB>

And excerpts of the script:

$script:CMDBInfo = [xml](gc D:\src\Enterprise\Enterprise\Systems\Scripts\Powershell-SCMDEV\Opsbrain\TMCMDB.xml)
$TargetNetwork = "$($json_data.environment)ServerPrivate"
$NetworkRanges = $script:CMDBInfo.CMDB.IPTable.Network | Where-Object {$_.Name -eq $TargetNetwork}
# Some ranges may be inactive, meaning we don't want to use them anymore
# Or some ranges may not have any available IP addresses, if multiple ranges are assigned to the environment then we'll want to try all of them to get a valid IP.
foreach($RangeInfo in $NetworkRanges.Range){
    if(-not ($RangeInfo.Active -eq 'false')){
        $IPAddress = GetIPFromIPAM -Subnet $RangeInfo.Subnet
        if([regex]::Matches($IPAddress,'([0-9]{1,3}\.){3}[0-9]').Success -eq 'True'){
            break
        }
    }
}
if([regex]::Matches($IPAddress,'([0-9]{1,3}\.){3}[0-9]').Success -eq 'True'){
    $script:json_data | Add-Member -Type NoteProperty -Name ip_address -Value $IPAddress
    $script:json_data | Add-Member -Type NoteProperty -Name opb_vm_port_group -Value $RangeInfo.VMwareNetworkLabel
    $script:json_data | Add-Member -Type NoteProperty -Name opb_netmask -Value $RangeInfo.SubnetMask
    $script:json_data | Add-Member -Type NoteProperty -Name opb_default_route -Value $RangeInfo.DefGW
    $script:json_data | Add-Member -Type NoteProperty -Name opb_name_server -Value $RangeInfo.DNS
} else {
    LLToLog -EventID $LLERROR -Text "Unable to get an IP address for $TargetNetwork"
    #Exit
}

$script:json_data

After dumping the $script:json_data at the end, all of the data is populated identically to the hardcoded values I was using in the previous (working) iteration of the script.

Then I use the data to populate the OSCustomization object:

$customization = Get-OSCustomizationSpec -Name "Spec_$host_name" | Get-OSCustomizationNicMapping | Set-OSCustomizationNicMapping  -IpMode UseStaticIP -IpAddress $script:json_data.ip_address -SubnetMask $script:json_data.opb_netmask -DefaultGateway $script:json_data.opb_default_route -Dns $script:json_data.opb_name_server[0],$script:json_data.opb_name_server[1]
$customization = Get-OSCustomizationSpec -Name "Spec_$host_name" | Set-OSCustomizationSpec -Domain $script:json_data.dns_domain -DomainUsername "service_account@domain.com" -DomainPassword $DomainAdminPwd

And then create the VM:

$create_vm = New-VM -Template $script:json_data.opb_template -ResourcePool $script:json_data.opb_build_cluster -Name $script:json_data.host_name -Datastore $datastore -DiskStorageFormat Thin -OSCustomizationSpec $customization -Confirm:$false -RunAsync -ErrorAction Stop

Right now, I have a Frankenstein of the old and new script. I attempt to do new-script stuff by pulling the information from the XML file, but then I ignore all of that and set the json_data values to the old-script hardcoded values. Therefore regardless of whether I read the XML or not, I'm always using the same values.

And if I comment out the read of the XML, the OSGuestCustomization finishes about 2 minutes after the New-VM clone finishes. If I leave it uncommented, the OSGuestCustomization never finishes. Some parts do, I think: the server believes it is joined to the domain, but the IP address doesn't get set (as reported by VMWare Tools), and the local admin account credentials don't get set.