How to Enable Evp Functions in Openssl

How to use openssl EVP library with low-level DES functions?

I think what you want to do should be possible. The EVP API is very much about using all parameters to initialize a context object, and then using the context object without any knowledge of what's inside it. You might even benefit from the fact that OpenSSL will let you supply parameters you don't need...such as supplying an IV (which will be ignored) when initializating a context for ECB mode.

Before going into the EVP version, lets talk a little about the functions you posted.

OpenSSL separates the production of a DES key into three steps:

Make the random bytes
Set/Correct the parity bits within the random bytes
Unroll the key schedule

The first step is done with any suitable (N/D)RBG (OpenSSL offers RAND_DRBG_bytes and RAND_bytes). The second step arises because some bits of a DES key do not contribute to the cipher strength; hence by convention they are set so as to give the bytes of the key odd parity. This is the function of DES_set_odd_parity (See https://crypto.stackexchange.com/questions/34199/purpose-of-des-parity-bits for more explanation.) The final step has a few options within OpenSSL: (a) trust the user to have given a good key, and unroll that; or (b) check what the user has provided to ensure it's not a weak key, and check that it has odd parity before unrolling the key schedule. The latter of these is performed by the function you list, DES_set_key_checked. The former, more trusting version, is DES_set_key_unchecked.

Okay, so why is all this useful? Well, it all has a direct analog in the EVP version of the API. With the DES functions, you would do the following:

Call RAND_bytes (or equivalent) to make a random key
Call DES_set_odd_parity to set the parity bits
Call DES_set_key_checked to unroll the key schedule
Call DES_<mode>_encrypt to encrypt using the unrolled key schedule

With EVP, you would do the following (mostly) equivalent steps:

Call EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_RAND_KEY, 0, dest_buf) to produce a random key appropriate to the cipher for which EVP_CIPHER_CTX *ctx has been initialized. This step also sets odd parity under the hood via a call to DES_set_odd_parity. Note that dest_buf is assumed to be large enough to hold the type of key that ctx is configured for. See line 280 in https://github.com/openssl/openssl/blob/master/crypto/evp/e_des3.c
Call EVP_EncryptInit passing the key, which calls DES_set_key_unchecked under the hood to unroll the key schedule and store it in the context. Note that this is the unchecked variant of set-key; the EVP API just assumes you're providing a good key. See line 226 in https://github.com/openssl/openssl/blob/master/crypto/evp/e_des3.c (for 2-key ede)

Note the minor chicken-and-egg problem: you have to init the context with the algorithm type, so that it knows what kind of key to make; and you have to supply the key to init the context. EVP_*Init gracefully handles NULL parameters, so it lets you partially initialize. E.g.,

uint8_t twokey[16];
EVP_EncryptInit(ctx, EVP_des_ede_cbc(), NULL, NULL);
EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_RAND_KEY, 0, twokey);
EVP_EncryptInit(ctx, NULL, twokey, iv);

Note also that you don't need OpenSSL to randomly generate the key if you already have one. EVP_EncryptInit is perfectly happy to trust whatever key you provide it.

Here's a rather long example using a hard-coded key, but using the openssl functions to correct its parity.

#include <openssl/evp.h>
#include <openssl/des.h>
#include <stdint.h>

int main(int argc, char *argv[])
{
  int res;
  uint8_t key[16] = {
    0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88,
    0x21, 0x32, 0x43, 0x54, 0x65, 0x76, 0x87, 0x98
  };

  uint8_t iv[8] = {
    0x12, 0x34, 0x56, 0x78, 0x9a, 0xbc, 0xde, 0xf0
  };

  uint8_t message[16] = {0,1,2,3,4,5,6,7,8,9,0xa,0xb,0xc,0xd,0xe,0xf};
  uint8_t ciphertext[24] = {0}; //leave room for padding!
  int ciphertext_total = 0;
  uint8_t decrypted [16] = {0};
  int decrypted_total = 0;

  //Select our cipher of choice: 2-key 3DES in CBC mode (hence the IV)
  const EVP_CIPHER *tdea_ede = EVP_des_ede_cbc();

  printf("hardcoded key:       ");
  {
    int i;
    for(i=0; i< (int)sizeof(key); i++)
    {
      printf("%02x ", key[i]);
    }
    printf("\n");
  }

  //Note I have to set parity on each of the keys, and I'm doing 2-key 3DES
  //DES_cblock is an annoying typdef of uchar[8]
  DES_set_odd_parity((DES_cblock *)key);
  DES_set_odd_parity((DES_cblock *)(key+8));

  printf("key with odd parity: ");
  {
    int i;
    for(i=0; i< (int)sizeof(key); i++)
    {
      printf("%02x ", key[i]);
    }
    printf("\n");
  }

  printf("Message:             ");
  {
    int i;
    for(i=0; i< (int)sizeof(message); i++)
    {
      printf("%02x ", message[i]);
    }
    printf("\n");
  }

  /* encrypt */
  {
    int outl = 0;
    EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
    EVP_EncryptInit(ctx, tdea_ede, key, iv);

    res = EVP_EncryptUpdate(ctx, ciphertext, &outl, message, (int)sizeof(message));
    printf("Update wrote %d bytes\n", outl);
    ciphertext_total += outl;

    EVP_EncryptFinal(ctx, ciphertext + ciphertext_total, &outl);
    printf("Final wrote %d bytes\n", outl);
    ciphertext_total += outl;
  }

  printf("Ciphertext:          ");
  {
    int i;
    for(i=0; i<ciphertext_total; i++)
    {
      printf("%02x ", ciphertext[i]);
    }
    printf("\n");
  }

  /* decrypt */
  {
    int outl = 0;
    EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
    EVP_DecryptInit(ctx, tdea_ede, key, iv);

    res = EVP_DecryptUpdate(ctx, decrypted, &outl, ciphertext, ciphertext_total );
    printf("Update wrote %d bytes\n", outl);
    decrypted_total += outl;

    EVP_DecryptFinal(ctx, decrypted + decrypted_total, &outl);
    printf("Final wrote %d bytes\n", outl);
    decrypted_total += outl;
  }

  printf("Decrypted:           ");
  {
    int i;
    for(i=0; i<decrypted_total; i++)
    {
      printf("%02x ", decrypted[i]);
    }
    printf("\n");
  }
}

Its output is:

$ ./a.out
hardcoded key:       11 22 33 44 55 66 77 88 21 32 43 54 65 76 87 98
key with odd parity: 10 23 32 45 54 67 76 89 20 32 43 54 64 76 86 98
Message:             00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
Update wrote 16 bytes
Final wrote 8 bytes
Ciphertext:          5d f9 44 ff 82 0b c3 47 90 be 11 fb 62 01 15 f0 65 45 f6 05 3f fa 81 96
Update wrote 16 bytes
Final wrote 0 bytes
Decrypted:           00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f

I'll admit, I was surprised to see the ciphertext was larger than the message, but then I remembered that OpenSSL applies padding by default.

If you're unfamiliar with the EVP API's init/update/final model, note that it's designed for streams. You can call update as many times as you like, and only once it's got enough data to process a block will you see it write any output. That's why "final" is there...to pad out any remaining data and process the final block if one is in flight.

Also, note that the example leaks the EVP_CIPHER_CTX objects...those should be freed at some point.

Working with EVP and OpenSSL, coding in C

For the sign/unsign key part I need further information, how is this signature done? For example, is this signature an X byte length at the end of the file and can then easily be removed?

For items 2-5 in your list the following code will surely assist, it is based on the examples from openssl documentation with more comments and adaptations for your needs. Feel free to ask if you have any questions that are not commented!

crpytor.c

#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <openssl/evp.h>

#define APPNAME "C"

#define CHUNK_SIZE 512
int do_crypt(FILE *in, FILE *out, int do_encrypt)
{
    /* Allow enough space in output buffer for additional block */
    unsigned char inbuf[CHUNK_SIZE];
    unsigned char outbuf[CHUNK_SIZE + EVP_MAX_BLOCK_LENGTH];
    int inlen;
    int outlen;
    EVP_CIPHER_CTX ctx;
    /* Bogus key and IV: we'd normally set these from
     * another source.
     */
    unsigned char key[] = { 0x13, 0xa3, 0xb4, 0xc1, 0x24, 0x19, 0xf5, 0x23, 0x18, 0xef, 0xca, 0x12, 0x4c, 0x9f, 0x14, 0xfe };
    unsigned char iv[] = { 0x92, 0x1c, 0x23, 0x3f, 0x5e, 0x10, 0x3d, 0x9a };
    /* Don't set key or IV because we will modify the parameters */
    EVP_CIPHER_CTX_init(&ctx);
    /* Using Blowfish encryption with cbc algorithm, you can use whichever is supported in openssl if you wish */
    EVP_CipherInit_ex(&ctx, EVP_bf_cbc(), NULL, NULL, NULL, do_encrypt);
    EVP_CIPHER_CTX_set_key_length(&ctx, 16);
    /* We finished modifying parameters so now we can set key and IV */
    EVP_CipherInit_ex(&ctx, NULL, NULL, key, iv, do_encrypt);
    for(;;)
    {
        inlen = fread(inbuf, 1, CHUNK_SIZE, in);
        if(inlen <= 0) break;
        if(!EVP_CipherUpdate(&ctx, outbuf, &outlen, inbuf, inlen))
        {
            /* Error */
            EVP_CIPHER_CTX_cleanup(&ctx);
            return -1;
        }
        fwrite(outbuf, 1, outlen, out);
    }
    if(!EVP_CipherFinal_ex(&ctx, outbuf, &outlen))
    {
        /* Error */
        EVP_CIPHER_CTX_cleanup(&ctx);
        return -1;
    }
    fwrite(outbuf, 1, outlen, out);
    EVP_CIPHER_CTX_cleanup(&ctx);
    rewind(in);
    rewind(out);
    return 0;
}

/* This is the standalone encryptor entry point */
int main(int argc, char** argv)
{
    FILE *encode_file;
    FILE *decode_file;
    int enc_or_dec;
    if (argc < 4)
    {
        printf("Usage: %s [plain file] [encrypted file] [0/1 deccrypt/encrypt]\n", argv[0]);
        return -1;
    }
    encode_file = fopen(argv[1], "r");
    decode_file = fopen(argv[2], "w+");
    /* Stupid decimal translation */
    enc_or_dec = *argv[3]-48;

    do_crypt(encode_file, decode_file, enc_or_dec);
    return 0;
}

And the Makefile:

all:
    gcc cryptor.c -o cryptor -g -lcrypto -I ../openssl-1.0.1f-host/include
clean:
    rm cryptor

This code does not use EVP_OpenInit() because it is used only for decryption, while my method (and your needs) require both encryption or decryption. While you can use EVP_OpenInit() to initialize a decryption context, I replaced the single call suitable only for decryption with two calls suitable for both encryption and decryption.

From the man page:

EVP_OpenInit() initializes a cipher context ctx for decryption with
cipher type. It decrypts the encrypted symmetric key of length ekl
bytes passed in the ek parameter using the private key priv. The IV
is supplied in the iv parameter. EVP_OpenUpdate() and EVP_OpenFinal()
have exactly the same properties as the EVP_DecryptUpdate() and
EVP_DecryptFinal() routines, as documented on the EVP_EncryptInit(3)
manual page.

EVP_OpenInit() for key files

If the signed file you are referring to is a public key file in RSA/DSA or similar format, you can use this StackOverflow question for a better method than mine as it does the key extraction from the file automatically (and uses EVP_OpenInit() like you require)

implementing openssl evp encryption

You're never actually setting iv to anything. Because iv is a local variable, its contents are uninitialized, and subsequently reading it results in undefined behavior.

What's probably happening when you call clock is that the stack space that was used by this function probably left some null bytes somewhere on the stack that just so happens to be where iv will reside when the encrypt function is called. You can't depend on this behavior, however.

Initialize iv to all zeros. Then you'll get the behavior you expect.

unsigned char iv[16] = { 0 };

OpenSSL RSA Encryption/Decryption with EVP Methods

With the help of @Topaco i found the solution. It is a Hybrid Encryption

I saved also the EncryptedKey and the IV to rsa_ek.bin and rsa_iv.txt file.
Decrypt the rsa_ek.bin with the openssl rsautl cli

openssl rsautl -decrypt -inkey RSAPrivateKey.pem -in rsa_ek.bin -out rsa_ek.txt

Decrypt the enc_msg.bin file with the AES-256-CBC Cipher

openssl enc -aes-256-cbc -d -in enc_msg.bin -K $(xxd -p -c 256 rsa_ek.txt) -iv $(xxd -p -c 256 rsa_iv.txt) -out enc_msg.dec.txt