CryptEncrypt AES 256 fails at encrypting last block

3

First of all, I apologize if I'm missing something here. This is my first every attempt at the Windows CryptAPI, and although I have gone through google, MSDN, MSDN's examples, etc. I cannot figure out why this problem is happening. I have the following code which is supposed to copy opcode (blocks of bytes) found at a specific address, encrypt them, and then write them back. It is by no means a complete code and it does very little error checking. I am using AES256.

bool CryptoClass::encrypt(DWORD address, DWORD len)
{
    DWORD dwBlockLen = 0; 
    DWORD dwBufferLen = 0; 
    DWORD dwCount = 0; 
    PBYTE pbBuffer = NULL;    
    dwBlockLen = AES_BLOCK_SIZE - AES_BLOCK_SIZE % ENCRYPT_BLOCK_SIZE; 
    dwBufferLen = dwBlockLen + ENCRYPT_BLOCK_SIZE; 
    if(pbBuffer = (BYTE *)malloc(dwBufferLen))
    {
        bool EOB = FALSE;
        while ( dwCount <= len) )
        {
            memcpy((void*)pbBuffer,(void*)address,dwBlockLen);              
            if ( (len - dwCount) < dwBlockLen) EOB = TRUE;              

            if(CryptEncrypt(hKey,NULL,EOB,0,pbBuffer,&dwBlockLen,dwBufferLen))
            {
                memcpy((void*)address,(void*)pbBuffer,dwBlockLen);
                address += dwBlockLen;                              
                dwCount += dwBlockLen;                              
            }
            else
            {
                error = GetLastError();
                MessageBoxA(NULL,"problem","error",MB_OK);
            }
        }
        free(pbBuffer);
       return true;
    }
    else return false;
}

From my understanding AES can encrypt blocks of 16 bytes, so AES_BLOCK_SIZE is defined to 16. Also my ENCRYPT_BLOCK_SIZE is set to 8. I am basically copying an example I found on MSDN, and adjusting for AES256 and for use with memory instead of a file. However in my while loop, whenever it reaches the end of the buffer, where the FINAL is set to TRUE, the CryptEncrypt fails. I have tried many different ways to do this but it always fails. Is it because the buffer at the end is less than 16bytes ? Can some please help, I am a total noob when it comes to encryption. Thank you EDIT: GetLastError returns: 0x000000ea

c++
winapi
encryption
asked on Stack Overflow Feb 1, 2012 by Ælex • edited Feb 1, 2012 by Ælex

3 Answers

4

You are getting the error ERROR_MORE_DATA. Which translates into:

If the buffer allocated for pbData is not large enough to hold the encrypted data, GetLastError returns ERROR_MORE_DATA and stores the required buffer size, in bytes, in the DWORD value pointed to by pdwDataLen.

If the method uses CBC encryption (Microsoft, in it's wisdom, does not specify a mode OR padding algorithm in their API) then the following algorithm should be performed to calculate the output size of the buffer:

buffer_size = (plain_size_bytes / AES_BLOCK_SIZE + 1) * AES_BLOCK_SIZE;

In other words, if plain_size_bytes is N * AES_BLOCK_SIZE you still need a full block of padding. This is because you always need to perform padding, otherwise the unpadding algorithm cannot distinquish you plain text from padding bytes.

Of course, you could also simply create a buffer of plain_size_bytes + AES_BLOCK_SIZE and use the pdwDataLen value to get the actual length.

EDIT: if you encrypt each plain text block separately, and the cipher uses CBC or ECB with padding, then the last plain block may require 2 whole blocks because of above

answered on Stack Overflow Feb 1, 2012 by Maarten Bodewes • edited Apr 30, 2014 by Community
1
while ( address < (address + len) )

My cryptic answer is "think about it -- you won't quite think about it forever, but..."

answered on Stack Overflow Feb 1, 2012 by Windows programmer
0
if ( (len - dwCount) < dwBlockLen) EOB = TRUE;

maybe should be something like

if ( (len - dwCount) < dwBlockLen)
{
    EOB = TRUE;
    dwBlockLen = ENCRYPT_BLOCK_SIZE;
}
answered on Stack Overflow Feb 1, 2012 by Windows programmer

User contributions licensed under CC BY-SA 3.0