CryptEncrypt fails with NTE_BAD_LEN

▲
1
▼

I need to encrypt message with X.509 RSA public key. I successfully imported 1024-bit public key in DER format, but my program fails with message longer than about 110 bytes. I'm calling CryptEncrypt function with pbData set to NULL, because I need to calculate size of output buffer first.

This is the source code:

screenshot

Plain text version of the same: 

bool CCrypt::RSAEncrypt() {
    HCRYPTPROV hProv = NULL;
    HCRYPTKEY  hKey = NULL;

    if (CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
        CParam *pubKey = coreData.local.overlay->getItem(3);

        // Decode the binary key blob in DER format into a CERT_PUBLIC_KEY_INFO
        CERT_PUBLIC_KEY_INFO* publicKeyInfo = NULL;
        DWORD publicKeyInfoSize;
        if (CryptDecodeObjectEx(
            X509_ASN_ENCODING,
            X509_PUBLIC_KEY_INFO,
            (LPBYTE) pubKey->getVal(),
            pubKey->getLength(),
            CRYPT_DECODE_ALLOC_FLAG/* | CRYPT_DECODE_NOCOPY_FLAG*/,
            NULL,   // TODO: Pass a CRYPT_DECODE_PARA to use own heap management to allocate memory
            &publicKeyInfo,
            &publicKeyInfoSize
        )) {
            // Import the public using the context
            if (CryptImportPublicKeyInfo(
                hProv,
                X509_ASN_ENCODING,
                publicKeyInfo,
                &hKey
            )) {
                // Get the size of a key
                DWORD dwBlockLen = NULL;
                DWORD dwValLen = sizeof(DWORD);
                if (CryptGetKeyParam(hKey, KP_BLOCKLEN, (LPBYTE) &dwBlockLen, &dwValLen, 0)) {
                    dwBlockLen = (dwBlockLen + 7) / 8;  // Transform to bytes legth
                    BYTE msg[] = "Lorem ipsum dolor sit .... [3000+ characters here]";
                    DWORD dwMsgLen = I(str)->lengthA((LPSTR) msg);
                    //dwMsgLen = 110;
                    DBG(C_INFO, "CryptGetKeyParam succeed. dwMsgLen: %d, dwBlockLen: %d", dwMsgLen, dwBlockLen);
                    // pbData [in, out] set to NULL to calculate actual size of a buffer required
                    if (CryptEncrypt(hKey, 0, TRUE, CRYPT_OAEP, NULL, &dwMsgLen, 0)) {
                        DBG(C_INFO, "CryptEncrypt succeed. dwMsgLen: %d", dwMsgLen);
                        // TODO: Fails here

                    } else {
                        DBG(C_ERROR, "CryptEncrypt error.");
                    }
                } else {
                    DBG(C_ERROR, "CryptGetKeyParam error.");
                }
                CryptDestroyKey(hKey);
            }
            LocalFree(publicKeyInfo);
        }
        CWA(advapi32, CryptReleaseContext)(hProv, 0);
    }

    return false;
}

Output from my debugger: 

[16:08:14] TC=1093889010, PID=25484(0x638C), TID=26340(0x66E4), LE=0(0x0), F=CCrypt::RSAEncrypt, FL=d:\c\source\client\../common/Crypt.cpp (62)
INFO: CryptGetKeyParam succeed. dwMsgLen: 2175, dwBlockLen: 128

[16:08:14] TC=1093889010, PID=25484(0x638C), TID=26340(0x66E4), LE=2148073476(0x80090004), F=CCrypt::RSAEncrypt, FL=d:\c\source\client\../common/Crypt.cpp (69)
ERROR: CryptEncrypt error.

As you can see, this function fails at CryptEncrypt with error NTE_BAD_LEN (0x80090004). However when I manually set dwMsgLen to 110 [see line 61], it works correctly. When I set this variable to 120, it fails.

Can someone tell me where the problem can be?

winapi
rsa
public-key-encryption
cryptoapi
x509
asked on Stack Overflow May 14, 2014 by Peter

0 Answers

Nobody has answered this question yet.

User contributions licensed under CC BY-SA 3.0