Windows service error during side-by-side COM activation: Unable to cast COM object of type 'System.__ComObject' to interface type

I have a side-by-side COM manifest, where my .NET desktop application works but the companion .NET Windows service gives the following error when trying to access the COM components:

Unable to cast COM object of type 'System.__ComObject' to interface type 'LibraryName.ComponentName'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{xxx}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).

Both projects are referencing the same Interop file and are installing with the same Interop, DLL, and manifest files. Both also have the same entry in their app.manifest files:

<dependentAssembly>
  <assemblyIdentity type="win32" name="LibraryName.sxs" version="2.15.1.0" />
</dependentAssembly>

What reasons would prevent a Windows service from running a side-by-side deployment, when a desktop application is able to run fine?