Splunk UniversalForwarder fails with " DetermineContextForAllProducts failed witht: 0x65b"

I am attempting to deploy the Splunk UniversalForwarder as an SCCM application using an MSI Deployment Type to a small group of testing servers and am encountering an uncharacteristically confusing issue with the MSI installer.

The Deployment Type itself is pretty straight forward. I uploaded the MSI file (splunkforwarder-6.2.1-245427-x64-release.msi) to the SCCM Site Server and let the Detection Method be auto-filled from the MSI. The only thing I did was change the install string to include the following arguments:

msiexec.exe /lv splunkinstall.log /i "splunkforwarder-6.2.1-245427-x64-release.msi" AGREETOLICENSE=YES DEPLOYMENT_SERVER="splunkd.security.contoso.com" /quiet /norestart /qn

Due to the inclusion of the /lv switch I can go to the C:\Windows\CCMcache folder and view the installation log:

GetPreviousSettings:  Error: DetermineContextForAllProducts failed witht: 0x65b.
GetPreviousSettings:  Error 0x80004005: Failed to GetInstalledSplunkSettings.
GetPreviousSettings:  Info: Leave GetPreviousSettings: 0x80004005.
CustomAction GetPreviousSettings returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
Action ended 14:20:23: GetPreviousSettings. Return value 3.

A little Google-fu and I find this posting over at Splunk's Community Support but I'm not terribly confident in correctness of the answers; however the failing installer step is GetInstalledSplunkSettings so maybe I should go digging through the registry to see if stale Product or Installer keys are erroneously causing the installer to try to uninstall nonexistent instances of UniversalForwarder.

With a combination of ProcMon and manually searching through the registry I managed to locate the following keys that seem to be related.

HKCR:\Installer\Products\B0271F4D65C5D084FA81634DC56AD4A
HKCR:\Installer\Features\B0271F4D65C5D084FA81634DC56AD4AE
HKCR:\Installer\UpgradeCodes\13631B46466632F4FA2E89CF8E9602DB

HKLM:\SOFTWARE\Classes\Installer\Features\B0271F4D65C5D084FA81634DC56AD4AE
HKLM:\SOFTWARE\Classes\Installer\UpgradeCode\13631B46466632F4FA2E89CF8E9602DB

ProcMon Events during installation:

10:17:31.8751924 AM MsiExec.exe 7436    RegQueryKey HKLM    SUCCESS Query: HandleTags, HandleTags: 0x0
10:17:31.8752103 AM MsiExec.exe 7436    RegOpenKey  HKLM\Software\Microsoft\Windows\CurrentVersion\Installer\Managed\S-1-5-18\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE   NAME NOT FOUND  Desired Access: Read
10:17:31.8752409 AM MsiExec.exe 7436    RegQueryKey HKU SUCCESS Query: HandleTags, HandleTags: 0x0
10:17:31.8752584 AM MsiExec.exe 7436    RegOpenKey  HKU\S-1-5-18\Software\Microsoft\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE REPARSE Desired Access: Read
10:17:31.8752831 AM MsiExec.exe 7436    RegOpenKey  HKU\.DEFAULT\Software\Microsoft\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE NAME NOT FOUND  Desired Access: Read
10:17:31.8753062 AM MsiExec.exe 7436    RegQueryKey HKLM    SUCCESS Query: HandleTags, HandleTags: 0x0
10:17:31.8753230 AM MsiExec.exe 7436    RegOpenKey  HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE    SUCCESS Desired Access: Read
10:17:31.8753486 AM MsiExec.exe 7436    RegQueryValue   HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE\ProductName    SUCCESS Type: REG_SZ, Length: 38, Data: UniversalForwarder
10:17:31.8753716 AM MsiExec.exe 7436    RegCloseKey HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE    SUCCESS 

10:34:31.5741168 AM MsiExec.exe 7560    RegEnumKey  HKCR\Installer\Products SUCCESS Index: 11, Name: B0271F4D65C5D084FA81634DC56AD4AE

10:34:31.5744153 AM MsiExec.exe 7560    RegOpenKey  HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE    SUCCESS Desired Access: Read
10:34:31.5744407 AM MsiExec.exe 7560    RegQueryValue   HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE\ProductName    SUCCESS Type: REG_SZ, Length: 38, Data: UniversalForwarder
10:34:31.5744637 AM MsiExec.exe 7560    RegCloseKey HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE    SUCCESS 

Now things get interesting! I remove the keys expecting success, run the Client Application Deployment and Evaluation Cycle and I get the same disappointing results. I doubled check the NTFS permissions for those keys and SYSTEM has Full Control which seems to disprove this answer indicating that the 0x80004005. Hmmmmm. What if I run the installation manually?

Same installer, same installation arguments (copy and pasted right out of AppEnforce.log), run from a Run As Administrator cmd.exe prompt and it fails with the same error at the same step. If I go back and remove the registry keys again, and re-run the installer from cmd.exe it works! LOL WUT?

Let's re-cap:

• If the registry keys are present the installer fails at the DetermineContextForAllProducts installation step both as run by the SCCM client and manually by me.
• If I remove the registry keys the SCCM client tries to run the installer and fails at the DetermineContextForAllProducts step.
• If I remove the registry keys and I manually run the same installer with the same arguments from a Run As Administrator cmd.exe prompt it SUCCEEDS! ¯\_(ツ)_/¯
• I get the same results on two very different servers.

Any ideas? I'm happy to provide a whole copy of my ProcMon events or any other information if that would be helpful.

Just heard back from support. This is a known issue in 6.2.2, and a fix is already in the pipeline for release in an upcoming version. I can't promise you a version number, but you can watch release notes for defect number SPL-95121.