Prevent custom actions from running during patch

I am authoring a very small patch to a very large package, it's sole purpose is to update a single file and add four smaller ones.

Using the WiX help as a guide I am able to generate the MSP file.

However, a patch runs apparently it runs the original package in reinstall mode, with all the custom actions and whatnot that go along with it, which is not what I want.

Further research turned up the OptimizeCA property of the MsiPatchMetadata table, and its WiX equivalent OptimizeCustomActions which allows custom actions to be skipped when applying a patch.

That sounded exactly what I wanted but unfortunately it did not work as expected. The original package has a bunch of XML config file change custom actions, and looking at the log it appears to be erroring out when it hits the XmlFile CAs:

MSI (s) (58!74) [14:53:24:928]: PROPERTY CHANGE: Adding ExecXmlFileRollback property. Its value is (stuff deleted)
MSI (s) (58:74) [14:53:24:928]: Doing action: ExecXmlFileRollback
Action 14:53:24: ExecXmlFileRollback. 
Action start 14:53:24: ExecXmlFileRollback.
MSI (s) (58:74) [14:53:24:928]: Skipping Action: ExecXmlFileRollback. It is being skipped as per the value provided for OptimizeCA in MsiPatchMetadata table of an applicable patch
Action ended 14:53:24: ExecXmlFileRollback. Return value 0.
SchedXmlFile:  Error 0x8007065a: Failed MsiDoAction on deferred action
SchedXmlFile:  Error 0x8007065a: failed to schedule ExecXmlFileRollback for file: (file)
SchedXmlFile:  Error 0x8007065a: failed to begin file change for file: (file)
Action ended 14:53:24: SchedXmlFile. Return value 3.

You can see that the OptimizeCA property is being respected for the ExecXmlFileRollback action, as it is being skipped, but something goes wrong when it tries to schedule it and that causes the whole thing to crash the installer.

This is the relevant part of my WiX patch authoring:

<OptimizeCustomActions 
  SkipAssignment="no"
  SkipImmediate="no"
  SkipDeferred="yes">
</OptimizeCustomActions>

I'm really having a hard time figuring out what to try next. I found another person having the exact same issue as me on wix-users but there was no response to the query.

A condition including "Not PATCH" should prevent the CA from running during patch install.

Alternatively, "Not Installed" would prevent it from running if the product is already installed, because it looks like you'd have a similar issue during a repair.