SChannel, SEC_E_ALGORITHM_MISMATCH (0x80090331)

Days of troubleshooting on this one, googling solutions & re-reading Microsoft documentation on the needed functions. Changing variables, retrying again and again. Help is very thoroughly appreciated, I'm sure it's not just me running into this.

I am working to implement networked client & server apps which communicate with a SSL/TLS layer using SChannel (and later will also get this working with OpenSSL for cross-compatibility). I have a client which is known to work, so we can focus on server-side.

For now the goal is to have it work without providing a certificate on either side (they should just generate on the fly as needed). I perform AcquireCredentialsHandle, load up the initial token from the client (running on the same host) and call AcceptSecurityContext.

It seems that no matter what variable I change I always end up with the same 0x80090331 error on the first call to AcceptSecurityContext. Tested on windows 7 & Windows Server 2012.

It seems to me there must be something outside my code, an OS setting that I need to fix. I'm finding contradictory information on the web. TLS 1.1 & TLS 1.2 have been added to the registry under SecurityProviders\SCHANNEL\Protocols* and set with a data DisabledByDefault=0. Also added ", schannel.dll" to 'SecurityProviders'.

Code is as follows:

... snip ... (Code to call AcquireCredentialsHandle)

PCCERT_CONTEXT serverCerts[1] = {0};

SCHANNEL_CRED sc = {0};
sc.dwVersion = SCHANNEL_CRED_VERSION;
//sc.grbitEnabledProtocols = SP_PROT_SSL3_SERVER | SP_PROT_TLS1_SERVER | SP_PROT_TLS1_1_SERVER | SP_PROT_TLS1_2_SERVER;
sc.grbitEnabledProtocols = SP_PROT_TLS1_1_SERVER | SP_PROT_TLS1_2_SERVER;
sc.dwFlags = 0;
sc.cCreds = 0; // Let Crypto API find the appropriate certificate for us
sc.paCred = serverCerts;

TimeStamp tsExpiry;
SECURITY_STATUS status = AcquireCredentialsHandle(
    NULL,
    UNISP_NAME,
    SECPKG_CRED_INBOUND,
    NULL,
    &sc,
    NULL,
    NULL,
    &hCredential,
    &tsExpiry);
std::cout << "AcquireCredentialsHandle result = 0x" << std::hex << status << std::endl;

... snip ... (Code to call AcceptSecurityContext)

    // TOKEN is received from client into m_receivedData
    //std::vector<char> m_receivedData;

    m_ctxtFlags = ASC_REQ_ALLOCATE_MEMORY | ASC_REQ_STREAM;
    SecBuffer inBuffers[2];

    // Provide Schannel with the remote host's handshake data
    inBuffers[0].pvBuffer    = (char*)(&m_receivedData[0]);
    inBuffers[0].cbBuffer    = (unsigned long)m_receivedData.size();
    inBuffers[0].BufferType  = SECBUFFER_TOKEN;

    inBuffers[1].pvBuffer   = NULL;
    inBuffers[1].cbBuffer   = 0;
    inBuffers[1].BufferType = SECBUFFER_EMPTY;

    SecBufferDesc inBufferDesc = {0};
    inBufferDesc.cBuffers   = 2;
    inBufferDesc.pBuffers   = inBuffers;
    inBufferDesc.ulVersion  = SECBUFFER_VERSION;

    SecBuffer outBuffers[2];

    // We let Schannel allocate the output buffer for us
    outBuffers[0].pvBuffer   = NULL;
    outBuffers[0].cbBuffer   = 0;
    outBuffers[0].BufferType = SECBUFFER_TOKEN;

    // Contains alert data if an alert is generated
    outBuffers[1].pvBuffer   = NULL;
    outBuffers[1].cbBuffer   = 0;
    outBuffers[1].BufferType = SECBUFFER_ALERT;

    SecBufferDesc outBufferDesc = {0};
    outBufferDesc.cBuffers   = 2;
    outBufferDesc.pBuffers   = outBuffers;
    outBufferDesc.ulVersion  = SECBUFFER_VERSION;

    unsigned long fContextAttr;
    TimeStamp tsTimeStamp;

    SECURITY_STATUS status = NULL;
    if (isFirstCall) {
        status = AcceptSecurityContext(
            &hCredential,
            NULL,
            &inBufferDesc,
            m_ctxtFlags,
            0,
            &hNewContext,
            &outBufferDesc,
            &fContextAttr,
            NULL);
        std::cout << "AcceptSecurityContext (isFirstCall) result = 0x" << std::hex << status << std::endl;
    }

SSL/TLS requires a server-side certificate. A missing server-side certificate generates the 0x80090331 error. The next step is to create a self-signed certificate as needed. See CertCreateSelfSignCertificate. A C example using CryptoAPI can be found here.