When one creates a new EKS cluster, the management plane returns the following bits of information to you:
The public URL of the Load-Balancer which talks to your 3 deployed masters (i.e. API server endpoint)
The Certificate Authority data key
Some other odds and ends, like the Cluster ARN, name, security groups, version and status
You can get #1 and #2 from the AWS CLI using the following commands:
What is the CA data key useful for?
It is common practices for Kubernetes clusters to self-signed their digital certificates.
I often get from Security Practictioners the hairy eyeball when this fact is discussed. Why not use “real” certificates, that are signed by trusted CAs. Well, some people do that, but for the rest of us, you are simply not gaining any signficant security benefits, and you are creating more work for yourself. You see, Kubernetes clusters use many digital certificates in all aspects of managaging a cluster. For example, each node has its own digital certificate to verify its authenticity. If you have a 100 nodes, are you going to register 100 “real” certificates - no, I don’t think so. Still, it would not shock me if in some future Kubernets releases the masters automatically generate a “real” digital certificate by leveraging Let’s Encrypt, for example. This may increase security at least a little bit, because let’s be honest… we all click through the browser warnings, don’t we?!
For the uber-paranoid, please see this blog post by Julia Evans, where she outlines at least 5 different ways of setting up a cluster using different CA strategies. These strategies won’t work for EKS or other managed Kubernetes clusters, but if you want to roll your own using KOPS, please check out her comments.
So, we ARE going to use a self-signed digitial certificate for the master nodes - get with the program. While all the tooling will automatically confirm that the certificates are valid, it would be nice if we could manually confirm it as well. That is what we are going to do in the rest of this blog post.
Get the certificates in the proper format
The public key provided by the AWS management plane is the signing key. It is NOT the key used in TLS certificates or anywhere else. Since it signs other keys, it is the root CA for the cluster. In order to use it, we must be aware it has been base64 formatted. Let’s undo that…
Also, if you examine the output below carefully, you will see the following line: CA:TRUE
Now, let’s grab the TLS certificate, used by the web server on the master nodes. I would simply export it from your browser, after you have gone to the API Server’s endpoint. You can also use openssl, like this:
Verify the certificate chain using openssl
Grab the certificate at the end of the output, and place it into a file.
I called mine kube-apiserver.pem. Now, let’s ask openssl to verify the TLS cert used by the API server has been signed by the CA given to us by the AWS management plane.
If you don’t see the OK, then you have a security issue. Otherwise, you are fine.
As mentioned, all the Kubernetes tooling does this for you automatically. If you examine your kubectl config file, you will notice that the CA certificate at the very top of the file. kubectl checks the certificate chain for you ever time. Thank you kubectl!!!
By understanding the Certificate Authority data key, given to us by the AWS management plane, we now have a better understanding of how TLS security works for self-signed hosts, or EKS in particular.
Now, some of you may be concerned that I just publically shared my CA certificate. While I do not recommend this as a best practice, my cluster is entirely safe. Why you ask? All AWS EKS clusters use IAM credentials along with standard Kubernetes RBAC controls. This feature, using external authorization plugins, is still a BETA feature in Kubernetes version 1.11, but has been fully adopted by AWS EKS. I will talk about it more in my next blog post.