How to decrypt data that is encrypted by OpenSSL with RSA_PKCS1_OAEP_PADDING using NCryptDecrypt?

I have difficulty to decrypt data being encrypted using OpenSSL and RSA_PKCS1_OAEP_PADDING padding option.

What I am doing is:

    BCRYPT_ALG_HANDLE hCryptAlg = NULL;
    BCRYPT_OAEP_PADDING_INFO paddingInfo = { 0 };
    DWORD cbDecryptedMessage;
    BYTE* pbDecryptedMessage = NULL;

    paddingInfo.pszAlgId = BCRYPT_SHA1_ALGORITHM;

    // Open an algorithm handle.
    BCryptOpenAlgorithmProvider(&hCryptAlg, BCRYPT_RSA_ALGORITHM, NULL, 0);

    // Calculate the required buffer 
    NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, NULL, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

    // After required buffer is allocated...
    NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, pbDecryptedMessage, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

It fails with NTE_INVALID_PARAMETER (0x80090027). I tried different flags but none of them works.

Note: m_hKeyContextFull has already been retrieved using CryptAcquireCertificatePrivateKey function call:

 m_hSystemStore = CertOpenStore(CERT_STORE_PROV_SYSTEM, 0, NULL, CERT_SYSTEM_STORE_LOCAL_MACHINE, m_storeName.c_str());

 m_pCertWithKeys = CertFindCertificateInStore(m_hSystemStore, SupportedEncodings, 0, CERT_FIND_SUBJECT_STR, m_certName.c_str(), NULL);

 // Obtain the private key from the certificate.
 DWORD m_KeyContextSpec = 0;
 HCRYPTPROV_OR_NCRYPT_KEY_HANDLE m_hKeyContextFull;
 CryptAcquireCertificatePrivateKey(m_pCertWithKeys, CRYPT_ACQUIRE_SILENT_FLAG | CRYPT_ACQUIRE_PREFER_NCRYPT_KEY_FLAG, NULL, &m_hKeyContextFull, &m_KeyContextSpec, &m_KeyContextMustBeReleased);

Note: All error checkings have been removed from code for readability.

Is there any idea what am I doing wrong?

Thanks.

I'd start by making paddingInfo valid:

BCRYPT_OAEP_PADDING_INFO paddingInfo;
paddingInfo.pszAlgId = BCRYPT_SHA1_ALGORITHM;
paddingInfo.pbLabel = NULL;
paddingInfo.cbLabel = 0;

The current standard has the empty string label (see RFC 3447).

Moreover, as to pbDecryptedMessage:

If this parameter is NULL, this function will calculate the size needed for the decrypted data and return the size in the location pointed to by the pcbResult parameter.

So it won't decrypt anyway. You need to allocate the buffer first to the right size; a lot of Windows API's work that way: first call it with output buffer NULL and you get back the needed size somehow, in this case in outputDataLen (which you should declare!). Then allocate the outbufbuffer to that size, and call the function again with the fresh buffer, and the correct length in cbDecryptedMessage. After use, free the buffer again of course. But your comment claims to have done so?

Another suspicious fact: You use NCryptDecrypt so the first argument, should be a hKey of the right type, while m_hKeyContextFull does not seem to have that type. CryptAcquireCertificatePrivateKey will get you an old style key handle. You cannot mix these different Windows CryptoAPI's. Maybe look at the NcryptImportKey function, to transfer it.