OpenSSL RSA library throws exception

I would like to include RSA encryption to my DLL project written in C++ using Visual Studio 2017, and I decided that OpenSSL would be perfect. I've added correct path to my OpenSSL folder in Additional Include Directories and Additional Library Directiories, I've added libcrypto.lib/libeay32.lib to Additional Dependencies. The project compiles, but when I want to use some functions from rsa.h like RSA_size, I'm receiving exception like:

Exception thrown at 0x0F17EB64 (libcrypto-1_1.dll) in Win32Project1.exe: 0xC0000005: Access violation reading location 0x00000004.

I've assumed that this is because some pointer went NULL in the library, but I can't find the reason. This told me to try this library with regular exe application, but the same error occurs even with the simplest code

#include <openssl/rsa.h>
int main()
{
    RSA *m_rsa = RSA_new();
    RSA_size(m_rsa);
    return 0;
}

I've tried to implement OpenSSL in version 1.1.0 and 1.0.2, both with same result.

RSA_new correctly allocates new RSA instance and ERR_get_error() returns "0".

Have a look at the library documentation:

If the allocation fails, RSA_new() returns NULL and sets an error code that can be obtained by ERR_get_error(3). Otherwise it returns a pointer to the newly allocated structure.

The RSA_new can fail an return a null pointer. Using RSA_size() will use this null pointer (address 0) to access a field of the RSA structure. If this field is at the offset 4, you will try to access the address 4 (base address 0 + offset 4) and you will get an "Access violation reading location 0x00000004".

So, check the returned pointer. If null, check the error code to see what happened...

Library links:

• RSA_new
• ERR_get_error
• ERR_error_string

Edit:

Another thing to know is that RSA->n should not be null when calling RSA_size() (see documentation). n is for the modulus, so you cannot call RSA_size() until you have generated or setted a key.

See this example (permalink here):

#include <openssl/rsa.h>
int main ()
{
    RSA* rsa = RSA_new ();
    std::cout << "rsa pointer = " << rsa << std::endl;
    std::cout << "rsa->n = " << rsa->n << std::endl;
    RSA_size ( rsa ); // Seg fault.

    return EXIT_SUCCESS;
}

Output:

rsa pointer = 0x12d9c20
rsa->n = 0
bash: line 7: 10149 Segmentation fault      (core dumped) ./a.out