Secrets?

Introduction

A Secret is a Kubernetes resource that contains sensitive data such as passwords, tokens, or keys. Storing these kinds of information in a Secret is safer and more flexible than putting it verbatim in a Pod.

Among other risks, a user who can create a Pod that uses a Secret can also see that Secret's value. Also, anyone with root permission on any node can read any Secret from the API server by impersonating the kubelet...

This post will cover how to protect the Secret content at rest when stored in the etcd database by modifying the kube-apiserver component. This protects the Secret value from being read, even if malicious users access the:

• etcd data directory, usually on /var/lib/etcd/member/snap/db
• etcd via etcdctl. Example:

$ ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get /registry/secrets/default/default-token-t7j4c

TL;DR

By default, Kubernetes Secrets are available in the etcd database in plain text. Use an EncryptionConfiguration YAML file to configure the encryption in the Kubernetes API server.

base64 encoded

If you start on Kubernetes, you could have the wrong impression about security while using Secrets because it appears to be base64 encoded.

Encoding modifies data from one format into another format using a scheme that is publicly available. ... Therefore the only difference between encoding and encryption is that encoding does not require a key. But, only the algorithm used to encode it. Yet encryption requires a key to reverse the content into plain text.

Source: All You Need To Know About Hashing, Encryption, Salting & Encoding

What it means is that anyone with enough privileges can discover Secret values from the cluster without problems.

Developer perspective

You, as a developer, create Kubernetes resources to deploy applications using Deployment, Ingress, Service, ConfigMaps, and Secrets (among many others).

You usually interact with kubectl to develop these resources. From the CLI perspective, you will see the base64 value of the Secret independently if it is encrypted in the database or is in plain text.

Note that your user experience is not affected by the encryption of the database.

Admin perspective

As the platform's admin, you can query the Kubernetes cluster directly via API, or you can also query the etcd data (using etcdctl or by reading the DB data files). So you can see and understand the difference between encrypted and unencrypted data and its value.

Usually, you will have to pass multiple security checks; having the database encrypted at rest will be one of them.

Enable encryption

To encrypt the database at rest, it is possible to configure the Kubernetes API server to encrypt the value of any generated/submitted Secret to the API server to send it encrypted to the database. So the configuration has to be done in the Kubernetes API server.

In this simple guide, you need to have access to the Kubernetes control plane node along with root permissions.

Create a new file in a new directory inside the Kubernetes default directory.

$ mkdir -p /etc/kubernetes/cypher
$ cat <<EOF >>/etc/kubernetes/cypher/config.yml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: $(head -c 32 /dev/urandom | base64)
    - identity: {}
EOF

Then modify your kube-apiserver static manifest to add the following configuration:

$ diff /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.original 
22d21
<     - --encryption-provider-config=/etc/kubernetes/cypher/config.yml
99,101d97
<     - mountPath: /etc/kubernetes/cypher
<       name: k8s-cypher
<       readOnly: true
129,132d124
<   - hostPath:
<       path: /etc/kubernetes/cypher
<       type: DirectoryOrCreate
<     name: k8s-cypher

These three diff allow you to:

• Mount the right host /etc/kubernetes/cypher directory  as a Pod volume k8s-cypher.
• Mount the right Pod volume k8s-cypher into the right container path: /etc/kubernetes/cypher.
• Add the --encryption-provider-config argument to the kube-apiserver container command.

After the change, the kube-apiserver Pod will restart. Once restarted, the API server will encrypt every new Secret on the fly before sending it to the etcd database.

Rotate secrets

If you read the section above:

... Once restarted, the API server will encrypt every new Secret on the fly before sending it to the etcd database.

But, what about already existing Secrets? You have to know that they remain as plain text in the etcd database. To rotate (and encrypt) all the Secrets, execute:

$ kubectl get secrets --all-namespaces -o json | kubectl replace -f -

The command above reads all Secrets and then updates them to apply server-side encryption.

Note: If an error occurs due to a conflicting write, retry the command. For larger clusters, you may wish to subdivide the Secrets by namespace or script an update.

Finishing

This is an essential topic when deploying and managing Kubernetes clusters. It's common to see clusters without this configuration set. This is due to the weak defaults most Kubernetes installers provide (I’m looking at you, kubeadm).

You can find more information about this topic:

• https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data
• Kubernetes Secret encryption - and two mistakes to avoid