SSPI negotiation failed WSTrustChannelFactory

This one has me for a while now, I am trying to build a console app that can call a .net web/wcf service SP, the first leg is to get a token from the idP (ADFS4.0) the pasted code was working fine for a whole day, at some point it stopped working with the following error:

SOAP security negotiation with 'https://adfs.domain.in/adfs/services/trust/13/windowsmixed' for target 'https://adfs.domain.in/adfs/services/trust/13/windowsmixed' failed. See inner exception for more details.

The inner error is:

The Security Support Provider Interface (SSPI) negotiation failed.
NativeErrorCode: 0x80090350 -> SEC_E_DOWNGRADE_DETECTED

I have tried /13/windows and /windowstransport as well as the endpoint.

private static GenericXmlSecurityToken RequestSecurityToken()
{
    // set up the ws-trust channel factory
    var factory = new Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannelFactory(new WindowsWSTrustBinding(
                SecurityMode.TransportWithMessageCredential), new EndpointAddress(new Uri("https://adfs.domain.in/adfs/services/trust/13/windowsmixed"), EndpointIdentity.CreateSpnIdentity("adfs@domain.in")));
    factory.TrustVersion = TrustVersion.WSTrust13;
    var rst = new RequestSecurityToken
    {
        RequestType = RequestTypes.Issue,
        KeyType = KeyTypes.Bearer,
        AppliesTo = new System.ServiceModel.EndpointAddress(endpoint_address)
    };
    // request token and return
    return factory.CreateChannel().Issue(rst) as GenericXmlSecurityToken;
}

In my case, for some reason, the ADFS was available over VPN but the AD based authentication bits are not happening over VPN. That's why SEC_E_DOWNGRADE_DETECTED is coming. In a regular non VPN environment things are good.

Also, another observation is once SAML token is generated over a regular enterprise network. Subsequent calls to generate the SAML token are going through as expected even on VPN.

So, if you see this error just check if the network you are in is part of the domain (and not public or private network), for SSPI negotiation.