How to use CloudHSM to implement mutual TLS ( client side ) in Ruby

I'm asking this question after reading all of CloudHSM topics on StackOverflow, Cryptography, Information Security and CloudHSM forum but couldn't find anything helpful. Any idea or code snippet is helpful.

We have a Ruby application that is requesting to a web server via X.509 certificates and we should generate/host private key inside CloudHSM.

I followed CloudHSM documentation step by step and configured TLS offloading via NGINX and Apache HTTPD to understand how it is working, now I'm working on mutual TLS with CloudHSM.

My web server requires client certificate, I can validate that via cURL:

curl --cert app-selfsigned.crt --key app-selfsigned.key  -k https://127.0.0.1/index.html

Also I can use this Ruby code to authenticate via certifications on disk:

require 'faraday'
require 'openssl'

def ssl_options
  cert_file = File.read "app-selfsigned.crt"
  key_file = File.read "app-selfsigned.key"
  ssl_options = {
    verify: false,
    client_cert: OpenSSL::X509::Certificate.new(cert_file),
    client_key: OpenSSL::PKey::RSA.new(key_file)
  }
end

def connection
  dest = "https://127.0.0.1/"
  connection = Faraday::Connection.new(dest, ssl: ssl_options)
  connection.get
end

irb -I . -r rubytest.rb
connection

cURL and Ruby test via certification on disk:

I need to host app-selfsigned.key key inside the CloudHSM, how can I do that?

1) Can I do that via CloudHSM OpenSSL Dynamic Engine? if yes, even after cloudhsm engine installation(/opt/cloudhsm/lib/libcloudhsm_openssl.so), should I load and install engine every time in my code?

2) Or should I use PKCS#11 via p11-kit or pkcs11-openssl packages and p11tool command or Ruby PKCS#11?

3) Should I add anything related to n3fips_password inside my Ruby application?

Here is Ruby code that I'm trying to use CloudHSM with it (I'm using FAKE PEM key instead of real private key to point to real private key with label nginx-selfsigned_imported_key inside CloudHSM):

require 'faraday'
require 'openssl'

def initialize_openssl
  key_label = "nginx-selfsigned_imported_key"
  # OpenSSL Engine:
  OpenSSL::Engine.load
  e = OpenSSL::Engine.by_id('cloudhsm')
  e.ctrl_cmd("SO_PATH", "/opt/cloudhsm/lib/libcloudhsm_openssl.so")
  e.ctrl_cmd("ID", "cloudhsm")
  e.ctrl_cmd("LOAD")
  e.load_private_key("CKA_LABEL=#{ key_label }")
end

def ssl_options
  cert_file = File.read "app-selfsigned.crt"
  key_file = File.read "app-selfsigned_fake_PEM.key"
  {
    verify: false,
    client_cert: OpenSSL::X509::Certificate.new(cert_file),
    client_key: OpenSSL::PKey::RSA.new(key_file)
  }
end

def connection
  dest = "https://127.0.0.1/"
  Faraday::Connection.new(dest, ssl: ssl_options)
end

def connect
  initialize_openssl
  c = connection
  c.get
end

irb -I . -r rubytest_cloudhsm.rb
initialize_openssl

but I get this error:

OpenSSL::Engine::EngineError: invalid cmd name
from /root/self-signed/app-selfsigned/rubytest_cloudhsm.rb:9:in `ctrl_cmd'
from /root/self-signed/app-selfsigned/rubytest_cloudhsm.rb:9:in `initialize_openssl'
from (irb):1
from /bin/irb:12:in `<main>'

Adding them line by line:

Ruby error with CloudHSM:

Debug logs

OpenSSL Dynamic Engine has been installed sucessfuly:

export n3fips_password=<Crypto User Username>:<CU Password>
openssl engine -tt cloudhsm
# (cloudhsm) CloudHSM hardware engine support
#       SDK Version: 2.03
# [ available ]
openssl engine -vvvv dynamic -pre SO_PATH:/opt/cloudhsm/lib/libcloudhsm_openssl.so -pre ID:cloudhsm -pre LOAD
# (dynamic) Dynamic engine loading support
# [Success]: SO_PATH:/opt/cloudhsm/lib/libcloudhsm_openssl.so
# [Success]: ID:cloudhsm
# [Success]: LOAD
# Loaded: (cloudhsm) CloudHSM hardware engine support
openssl speed -engine cloudhsm
# SDK Version: 2.03
# engine "cloudhsm" set.
# Doing md2 for 3s on 16 size blocks:
# 557992 md2's in 2.99s
openssl version
# OpenSSL 1.0.2k-fips  26 Jan 2017
rpm -qa | grep -i openssl
# openssl-1.0.2k-16.amzn2.1.1.x86_64
# openssl-libs-1.0.2k-16.amzn2.1.1.x86_64

OpenSSL CloudHSM Dynamic Engine shared object is in the correct place:

ls -ltrha /usr/lib64/openssl/engines/libcloudhsm.so
lrwxrwxrwx 1 root root 40 Aug  7 09:56 /usr/lib64/openssl/engines/libcloudhsm.so -> /opt/cloudhsm/lib/libcloudhsm_openssl.so

libcloudhsm_openssl.so:

OS:

cat /etc/os-release
# NAME="Amazon Linux"
# VERSION="2"
# ID="amzn"
# ID_LIKE="centos rhel fedora"
# VERSION_ID="2"
# PRETTY_NAME="Amazon Linux 2"
# ANSI_COLOR="0;33"
# CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
# HOME_URL="https://amazonlinux.com/"
uname -a
# Linux hsm.example.net 4.14.133-113.112.amzn2.x86_64 #1 SMP Tue Jul 30 18:29:50 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

rpm -qa | grep -i cloudhsm-client
# cloudhsm-client-2.0.3-3.el7.x86_64
# cloudhsm-client-dyn-2.0.3-3.el6.x86_64

I can check private and public key with or without CloudHSM engines:

app-selfsigned.crt: Public Key
app-selfsigned.key: Private key has been exported from CloudHSM 
app-selfsigned_fake_PEM.key: Fake private key pointing to real private key inside CloudHSM generated by getCaviumPrivKey -k 14 -out app-selfsigned_fake_PEM.key

# Test without CloudHSM:
openssl s_server -cert app-selfsigned.crt -key app-selfsigned.key
# Using default temp DH parameters
# ACCEPT
openssl s_server -cert app-selfsigned.crt -key app-selfsigned_fake_PEM.key
# Using default temp DH parameters
# ACCEPT


# Test with CloudHSM engine
openssl s_server -cert app-selfsigned.crt -key app-selfsigned.key -engine cloudhsm
# SDK Version: 2.03
# engine "cloudhsm" set.
# Using default temp DH parameters
# ACCEPT
openssl s_server -cert app-selfsigned.crt -key app-selfsigned_fake_PEM.key -engine cloudhsm
# SDK Version: 2.03
# engine "cloudhsm" set.
# Using default temp DH parameters
# ACCEPT

To make sure I'm requesting the correct key inside CloudHSM, I configured app-selfsigned.crt and app-selfsigned_fake_PEM.key for TLS offloading via NGINX:

/etc/nginx/nginx.conf:

ssl_engine cloudhsm;
ssl_certificate "/etc/pki/nginx/app-selfsigned.crt";
ssl_certificate_key "/etc/pki/nginx/private/app-selfsigned_fake_PEM.key";

nginx -t
# SDK Version: 2.03
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful
systemctl start nginx && systemctl status nginx
# Aug 13 11:21:33 hsm.example.net nginx[13046]: SDK Version: 2.03

Check cert file via OpenSSL:

openssl x509 -in app-selfsigned.crt -text -noout
# Serial Number: c7:c4:07:a6:78:22:2e:ff
# Subject: C=AA, ST=AA, L=AA, O=AA, OU=AA, CN=a.com/emailAddress=a@a.com

Certificate check via Firefox:

Checking private key inside CloudHSM via key_mgmt_util and cloudhsm_mgmt_util:

/opt/cloudhsm/bin/key_mgmt_util
loginHSM -u CU -s CUADMIN -p CUPASSWORD
findSingleKey -k 14
# Cfm3FindSingleKey returned: 0x00 : HSM Return: SUCCESS
getKeyInfo -k 14
# Cfm3GetKey returned: 0x00 : HSM Return: SUCCESS
# Owned by user: 6

/opt/cloudhsm/bin/cloudhsm_mgmt_util /opt/cloudhsm/etc/cloudhsm_mgmt_util.cfg
enable_e2e
loginHSM CU CUADMIN CUPASSWORD

getAttribute 14 0
# OBJ_ATTR_CLASS
# 0x00000003
# 3: Private key in a public–private key pair. 
getAttribute 14 2
# OBJ_ATTR_PRIVATE
# 0x00000001
# 1: True. This attribute indicates whether unauthenticated users can list the attributes of the key. Since the CloudHSM PKCS#11 provider currently does not support public sessions, all keys (including public keys in a public-private key pair) have this attribute set to 1.
getAttribute 14 3
# OBJ_ATTR_LABEL
# nginx-selfsigned_imported_key
getAttribute 14 256
# OBJ_ATTR_KEY_TYPE
# 0x00000000
# 0: RSA. 

If you are using CloudHSM mutual TLS in another language please paste your code here so I can get the idea and implement it in Ruby.

Thanks in advance.

To anyone who see this question later, here is a sample Ruby code works with Amazon CloudHSM:

require 'openssl'
require 'base64'

FAKE_KEY = "/root/ruby/ruby_key_inside_hsm/ruby_hsm_fake_private.key"
REAL_KEY = "/root/ruby/ruby_key_inside_hsm/ruby_hsm_real_private_exported.key"
PUB_KEY = "/root/ruby/ruby_key_inside_hsm/pubkey.pem"

STR = "test string"

def encrypt(str)
  pubkey = OpenSSL::PKey::RSA.new(File.read(PUB_KEY))
  Base64.encode64(pubkey.public_encrypt(str))
end

def decrypt(str, key)
  OpenSSL::Engine.load
  privkey = OpenSSL::PKey::RSA.new(File.read(key))
  privkey.private_decrypt(Base64.decode64(str))
end


def estr
  encrypt(STR)
end

def real_dec
  decrypt(estr, REAL_KEY)
end

def hsm_dec
  OpenSSL::Engine.load
  OpenSSL::Engine.by_id('cloudhsm')
  decrypt(estr, FAKE_KEY)
end

Right now we are working on it to add it to the production environment.