Loading a client certificate in asp-net core fails on Windows Server 2012 R2

I have an ASP.NET Core 2.2 app that loads a client certificate to use when connecting to an external API. The code works perfectly when run locally using visual studio, but fails when run on our server.

I get the following error in the log:

2019-03-12 13:29:12.810 +01:00 [ERR] Connection id "0HLL6SPAO039S", Request id "0HLL6SPAO039S:00000001": An unhandled exception was thrown by the application. Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Unknown error (0xc0000225) at Internal.Cryptography.Pal.CertificatePal.FilterPFXStore(Byte[] rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags) at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(Byte[] rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags) at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags) at Geomatic.TinglysningApi.Startup.GetClientCertificate() in C:\dev\TinglysningApi\Geomatic.TinglysningApi\Geomatic.TinglysningApi\Startup.cs:line 49 at Geomatic.TinglysningApi.Startup.b__4_0() in C:\dev\TinglysningApi\Geomatic.TinglysningApi\Geomatic.TinglysningApi\Startup.cs:line 35

The code that loads the certificate looks like this

    private X509Certificate GetClientCertificate()
    {
        return new X509Certificate2(path, password, X509KeyStorageFlags.EphemeralKeySet);
    }

I have also tried with other X509KeyStorageFlags, and combinations of those but they all fail in the same way.

On the server I tried running my app as a IIS hosted app and I have tried running it from cmd using my own user and with cmd with admin permissions.