c - Error: "invalid use of incomplete type ‘RSA {aka struct rsa_st}" in OpenSSL 1.1.0

Question

I have old code that was written to link against an old version of openssl. Part of this code loads a key from a PEM file, and tries to understand whether this key is a private or public key, by using the following code:

if( (prv->p==0 || prv->q==0) ) {
    // This is not a private key!
    throw error("No private key for decryption");
}

With the latest version of openssl, this (justifiably) doesn't compile:

crypto.cpp: In function ‘key* decrypt_header(file_t, RSA*)’:
crypto.cpp:158:13: error: invalid use of incomplete type ‘RSA {aka struct rsa_st}’
     if( (prv->p==0 || prv->q==0) ) {
             ^~

I understand that direct access to the struct's private members was replaced with a function, but I am having a hard time figuring out which function that is.

Answer 1

crypto.cpp:158:13: error: invalid use of incomplete type ‘RSA {aka struct rsa_st}’
     if( (prv->p==0 || prv->q==0) ) {
             ^~

As you are aware, OpenSSL 1.1.0 changed the visibility of a lot of struct members. You can no longer access the members directly. Instead, you have to use getter and setter functions.

Try RSA_get0_factors. The get0 means the reference counts are not incremented. Do not BN_free them.

void RSA_get0_factors(const RSA *r, const BIGNUM **p, const BIGNUM **q);

---

If the code supports multiple versions of OpenSSL, then you will need a guard because RSA_get0_factors is for OpenSSL 1.1.0 and above. Maybe something like the following. Also see OPENSSL_VERSION_NUMBER man page.

#include <openssl/opensslv.h>

#if OPENSSL_VERSION_NUMBER < 0x10100000L

    /* OpenSSL 1.0.2 and below (old code) */

#else

    /* OpenSSL 1.1.0 and above (new code) */

#endif