Getting exception on adding SSL binding to IIS site through code "A specified logon session does not exist. It may already have been terminated."

I have a scenario where I need to start an IIS Site with SSL binding on a code condition. Below the code snippet

ServerManager serverManager = new ServerManager();
var webService = serverManager.Sites["webService"];
if (webService != null)
{
    serverManager.Sites.Remove(webService);
}

serverManager.Sites.Add("webService", "http", "*:8000:", phyPath);
serverManager.Sites["webService"].Bindings.Add("*:8001:", certificate.GetCertHash(), "My");
serverManager.CommitChanges();

It all executes well if I run it on http. But when adding SSL binding with certificate it throws this exception.

"A specified logon session does not exist. It may already have been terminated. (Exception from HRESULT: 0x80070520)"

Reading about this error, somewhere I read that it shouldn't be present in personal store and IIS - Server certificates, so I made sure it's not present before running the code. It also didn't work.

For testing I am using a self-signed certificate which I don't think should be an issue. Also didnt find any such thing when searching about this error. This certificate is a .cer certificate.