How to communicate with gRPC server hosted on Google Compute Engine over SSL/TSL?

Context

I have developed a gRPC server in Java and a corresponding gRPC client in C#. The objective is to call the gRPC server from several gRPC clients deployed on Windows machines.

Having looked at how gRPC is supported in Azure, AWS, and the Google Cloud Platform (GCP), I will likely host the gRPC server on GCP. Therefore, I am currently testing the deployment scenario for the gRPC server as described by Google in the tutorial on gRPC on Compute Engine. In short words, this means the gRPC server runs in a custom-built Docker container on a Google Compute Engine (GCE) Virtual Machine (VM), right next to the Extensible Service Proxy (ESP), which runs in its own, preconfigured Docker container on the same VM.

An important aspect for use in production is the ability to establish a secure communication channel between the gRPC clients and the gRPC server, using SSL/TSL. This is where I am having problems in the cloud hosting scenario (but not in the self-hosting scenario, where this works nicely).

What works so far?

The gRPC client, which runs on my local Windows 10 machine, communicates successfully with the gRPC server:

• over a secure SSL/TLS channel in case I am self-hosting the server on my local Windows 10 machine; and

• over an insecure channel in case I am hosting the server on GCE as described above.

I've issued the following commands on the GCE VM to create the docker containers for the successful client-server communication over the insecure channel.

# Create the container network.
sudo docker network create --driver bridge esp_net

# Create dss-signer container from docker image.
# The Java gRPC service listens on port 50051 (see esp container below).
sudo docker run \
--detach \
--name=dss-signer \
--net=esp_net \
gcr.io/[my-project-name]/dss-signer:1.1

# Create Extensible Service Proxy (ESP) container from predefined docker image.
# The ESP container's port 9000 is published as port 80 on the host machine,
# meaning a client will have to connect to port 80 on the host machine.
sudo docker run \
--detach \
--name=esp \
--publish 80:9000 \
--net esp_net \
gcr.io/endpoints-release/endpoints-runtime:1 \
--service=signer.endpoints.[my-project-name].cloud.goog \
--rollout_strategy=managed \
--http2_port=9000 \
--backend=grpc://dss-signer:50051

Thus, I'd say it works in general and there are no issues in the Java or C# code per se (outside of configuration-related issues related to making it work over SSL/TLS).

What does not work?

I've been unsuccessful in establishing a secure channel between the gRPC client and server, following the description on Enabling SSL. My C# client always throws the following exception:

Grpc.Core.RpcException
  HResult=0x80131500
  Message=Status(StatusCode=Unavailable, Detail="failed to connect to all addresses")
  Source=mscorlib
  StackTrace:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
   at SignerClient.AbstractSignatureHandler.<InitiateCall>d__4.MoveNext()

[Rest of stack trace removed as it did not contain any helpful hints.]

What have I tried?

Here's the openssl command used to create the server key and certificate. For the Common Name, I am using the GCE VM's IP address as displayed in the Google Cloud Console. I've copied the key and certificate to the /etc/nginx/ssl directory.

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./nginx.key -out ./nginx.crt -subj "/O=[My Org]/OU=Servers/CN=[GCE VM IP Address]"

Here's the docker command for creating and starting the ESP docker container, which, based on my understanding is what needs to change for enabling SSL/TLS. This also means I have not changed the "backend" gRPC server compared to what I described above. Not sure whether that is correct.

sudo docker run \
--detach \
--name=esp \
--publish 443:443 \
--net esp_net \
--volume=/etc/nginx/ssl:/etc/nginx/ssl \
gcr.io/endpoints-release/endpoints-runtime:1 \
--service=signer.endpoints.[my-project-name].cloud.goog \
--rollout_strategy=managed \
--ssl_port=443 \
--backend=grpc://dss-signer:50051

Here is the C# code used to set up the secure channel on the client's side:

const string host = "[GCE VM IP Address]";
const int port = 443;

// Create the SSL credentials.
string caCertPem = File.ReadAllText("Certs\\ca.cer");
string clientCertPem = File.ReadAllText("Certs\\client.cer");
string clientKeyPem = File.ReadAllText("Certs\\client.key");

var keyCertificatePair = new KeyCertificatePair(clientCertPem, clientKeyPem);
var sslCredentials = new SslCredentials(caCertPem, keyCertificatePair);

// Create a client that communicates over a secure channel.
var channel = new Channel(host, port, sslCredentials);

I've also tried variants of the above docker command, e.g., using a different SSL port (8080) or retaining the http2_port setting. Unfortunately, nothing has worked.

Thus, how do I set this up to have the gRPC client and GCE-hosted server communicate securely over SSL/TSL? Do I need to configure the C# client and Java server differently? How do I need to configure the ESP docker container?

Based on a helpful hint from Wayne Zhang in the Google group on Google Cloud Endpoints on enabling gRPC logging for the gRPC client and more research related to the error reported in the log, I found the answer to my own question.

How did I enable the gRPC log on the client?

To enable gRPC logging on the client running on my Windows 10 machine, I set the GRPC_TRACE and GRPC_VERBOSITY environment variables as follows:

set GRPC_TRACE=all
set GRPC_VERBOSITY=DEBUG

Running the gRPC client in a command prompt produced very verbose output with enough information to spot the issue.

What caused the error?

I was surprised to find an SSL handshake error in the log. This was due to the fact that the server certificate created as described in Google's how-to guide did not contain the subjectAltName X.509 extension. The .NET gRPC client (written in C#) seemingly expects that, although this never was an issue in my self-hosted scenario.

Google's how-to guide is also silent about that for two reasons. Firstly, the Extensible Service Proxy (ESP) documentation is primarily focused on OpenAPI and not gRPC. Secondly, it is language-agnostic and does not address language-specific issues like this one.

How did I solve the issue?

In a first attempt, I created a self-signed X.509 certificate with a common name (CN) and subjectAltName extension that were both set to the IP address of my GCE VM. However, that still did not work because the .NET gRPC client did not accept a self-signed server certificate (i.e., one where subject and issuer are identical). Therefore, I had to create a server certificate request and then create the server certificate in a second step, using my self-signed root CA certificate and key.

So, here's how I created the gRPC server certificate:

REM Create GCE server certificate in PEM encoding.
REM Set ipAddress to whatever IP address the GCE VM is using.

set ipAddress=[GCE VM IP address]
set subject=/O=[MyOrganization]/OU=[MyOrganizationalUnit]/CN=%ipAddress%
set subjectAltNameConfig=subjectAltName = IP:%ipAddress%
echo %subjectAltNameConfig% > extfile.cfg

openssl req -newkey rsa:2048 -keyout Certs\nginx.key -nodes -out Certs\Requests\nginx.csr -subj "%subject%" -addext "%subjectAltNameConfig%"

REM Sign GCE server certificate.

openssl x509 -req -extfile extfile.cfg -in Certs\Requests\nginx.csr -CA Certs\ca.cer -CAkey Certs\ca.key -passin pass:[password] -days 365 -set_serial 01 -out Certs\nginx.crt

Everything else in my code (Java gRPC server, C# gRPC client) and configuration (Docker commands for gRPC server and ESP with SSL enabled) was correct. The two files nginx.crt and nginx.key must be transferred to the GCE VM and stored in the folder /etc/nginx/ssl, which is mounted in the Docker command.