WIX installer execute vbscript from CustomAction

I can't execute VBScript using WIX installer. Currently I have this part of WiX config:

<Binary Id='KillThatProcessBinary' SourceFile='KillThatProcess.vbs' />
 <CustomAction Id="KillThatProcessAction"
               Execute="immediate"
               BinaryKey='KillThatProcessBinary'
               VBScriptCall='KillThatProcessFunction'
               Return="check"/>

<InstallExecuteSequence>
    <Custom Action='KillThatProcessAction' Before='InstallValidate'/>
    <ScheduleReboot After="InstallFinalize"/>
</InstallExecuteSequence>

And this VBS script (KillThatProcess.vbs):

Public Function KillThatProcessFunction()
  Set oShell = WScript.CreateObject("WSCript.shell")
  oShell.run "cmd /C wmic process where ""name like '%java%'"" delete"
  Return 0
End Function

I already try to insert this script into CustomAction (as innerText), and add attribute: Script="vbscript". But nothing works, every time I got the error message - "There is a problem with this Windows Installer package. A script reqired for this install to complete could not be run. Contact your support personnel or package vendor."

And errors in log file:

Error 0x80070643: Failed to install MSI package.
[1A88:2FA4][2018-08-21T14:11:17]e000: Error 0x80070643: Failed to configure per-user MSI package.
[1A88:2FA4][2018-08-21T14:11:17]i319: Applied execute package: LPGateway, result: 0x80070643, restart: None
[1A88:2FA4][2018-08-21T14:11:17]e000: Error 0x80070643: Failed to execute MSI package.

I already execute this vbs script (not from installer) and it works. Anybody know what I do wrong?

There are a few issues I want to summarize:

1. VBA & VBScript Functions: That VBScript looks like it is actually VBA and calling VBScript in an MSI requires a bit of tweaking to call VBScript functions properly.
2. Reboot: The reboot you schedule must get a better condition to avoid unexpected reboots.
3. Process Kill: What process are you trying to kill?
   • Elevation: If it is elevated you need to run the kill elevated for it to succeed. Your per-user setup is likely not set to elevate at all (so you can generally only end processes running as yourself).
   • Restart Manager: Very often you do not need to kill processes, due to the Restart Manager feature of Windows that Windows Installer tries to use. Attempt to explain this feature (look for yellow sections).

Issue 1: That must be a VBA script and not a VBScript? For the record: I haven't seen return in VBScript? In VBScript you return from a function by setting the function name equal to whatever you want to return, quick sample:

result = IsEmptyString("")
MsgBox CStr(result)

Function IsEmptyString(str)

  If str = "" Then 
     IsEmptyString = True 
   Else 
     IsEmptyString = False
  End If

End Function

Note: The above is just a silly, rather meaningless example. For more elaborate checking try IsBlank from ss64.com. VBScript comes with the functions IsEmpty and IsNull and IsObject.

When used in MSI files, I normally don't add a function in the VBScript, but just run the script directly, so running this VBScript should work:

MsgBox(Session.Property("ProductName"))

Inserting it into the WiX source (notice no function call specified):

<Binary Id='Sample.vbs' SourceFile='Sample.vbs' />
<CustomAction Id='Sample.vbs' VBScriptCall='' BinaryKey='Sample.vbs' Execute='immediate' Return='ignore'/>

Crucially your VBScript can still call other functions available in the same VBScript file. So in the above sample "IsEmptyString" can be called from the main "nameless" function at the top of the file.

Check Exit Code: And finally, any custom action set to check exit code can throw your setup into abort (immediate mode) or rollback (deferred mode). I would only check the exit code if you have to end the setup if the custom action can not run.

Issue 2: Reboots. This is a very serious issue in my opinion. I have seen people sent out the door for causing unexpected reboots during large scale deployments. Rebooting a knowledge worker's PC (and their managers) with a dozen Visual Studio windows open, dozens of browser windows and Word and Excel and you name it. It can cause a great deal of problems. And they may know where you live! :-)

Please read the following answer (at least its 3 bullet points at the beginning): Reboot on install, Don't reboot on uninstall

Issue 3: Process Kill. As indicated above the killing of processes is related to the reboot issues. It is not always necessary to kill processes if they are compliant with Windows Restart Manager as explained in the link above. Let me repeat it here (yellow sections should give you the gist of it - especially the second one I think).

There are a few different ways to kill processes. Note that the most common problem might be that you don't have the access rights and / or privileges to kill the process - no matter what tool or approach you use to do so.

Perhaps you can try the CloseApplication feature from the Util schema: http://wixtoolset.org/documentation/manual/v3/xsd/util/closeapplication.html

• In Wix MSI: Killing a process upon uninstallation
• Kill windows service forcefully in WIX (towards bottom)
• Kill process using WMI / VBScript
• Some people combine taskill.exe and WiX by means of the Quiet Execution Custom Action (CAQuietExec: hide command line windows). FireGiant documentation.

I am not sure which of these options to recommend. I don't like the concept of killing processes altogether, but sometimes there is no other option I guess.