Keeping Kubernetes Secrets secret in Flux
Managing Secrets is a tough challenge in GitOps. This article explores how Bitnami Sealed Secrets can be used to handle your Kubernetes Secrets in a Flux workflow.
The GitOps paradigm has revolutionized continuous deployment in Kubernetes. The basic idea behind GitOps is to manage resources on Kubernetes solely by interacting with Git repositories. This process requires the declarative description of our system's desired system in Git. Then GitOps operators, like Flux or ArgoCD, automatically and constantly synchronize the resources in the repository (the desired state) with those present in the cluster (the actual state).
This approach poses a severe challenge to Secrets management, as Secrets don't enjoy being exposed in Git repositories. The main issue is that Kubernetes Secrets are not really secret. The sensitive data they contain is stored as unencrypted base64-encoded strings (there is a great article that goes into more depth on Secrets). Therefore, they can't be placed directly in a repository and enjoy the automagical GitOps syncing process as other resources would.
Wait...what if I use a private repository?
Even if you use a private repository with Area51-like access control, it is still a very, very, very bad practice to store Secrets in repositories.
What then?
Luckily, this problem is not new. It was present long before the rise of the GitOps era and various tools have been proposed to address this issue.
The solutions mainly fall into two camps:
- Camp #1: Store Secrets outside the repository in an external, centralized place, aka "no secrets, no problem".
- Camp #2: Store encrypted Secrets in the repository, aka "encrypt all the things".
This articles explores Camp #2. It is a nice, decentralized solution that fits well in the GitOps workflow. In particular, we use Flux as a GitOps operator and Bitnami Sealed Secret to encrypt Kubernetes Secrets so that they can be safely stored in Git.
What is a Bitnami Sealed Secret?
A Sealed Secret is a custom resource (yeeee, yet another custom resource 🥳) that wraps a Kubernetes Secret, encrypting its content.
An example of a Sealed Secret is the following:
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: mysecret
spec:
encryptedData:
secret: AgBy3+2jJJSWK+PiTySYZ23xSJDEq.....The content of the spec.encryptedData.secret is now encrypted and it is safe to store in Git repositories (even public ones) and enjoy the GitOps magic.
Sealed Secrets can be decrypted only by a Sealed Secret controller (yeeee, yet another custom controller 🥳) that must be deployed in our Kubernetes cluster.
Sealed Secrets are a straightforward application of asymmetric cryptography. The Sealed Secret controller generates a public/private RSA key at startup. A user encrypts Secrets using the public key, and the controller decrypts them with the associated private key.
Sealed Secrets are usually referred to as one-way (or write-only) Secrets, since every client can create them (assuming access to the controller's public key), but only the controller can actually decrypt them and recover the original Kubernetes Secrets.
Deploy and use Bitnami Sealed Secrets in Flux
Assuming you already have a Kubernetes cluster synchronized with a repository, to use Sealed Secrets you need to:
- Install
kubeseal - Deploy (via GitOps, of course) the Sealed Secrets controller
- Retrieve the public key from the Sealed Secrets controller
- Use the public key to encrypt a regular Kubernetes Secret object into a Sealed Secret.
- Commit the Sealed Secret to the repository
- Wait for the Sealed Secret to be deployed and decrypted by the Sealed Secrets controller
A representation of the Sealed Secrets workflow in Flux is the following:
If you don't have a Kubernetes cluster at hand, a complete example of Bitnami Sealed Secrets in Flux that starts from scratch can be found here.
1. Install Kubeseal
kubeseal is a client-side command-line tool to encrypt Kubernetes Secrets into a SealedSecret.
Install the kubeseal CLI using Homebrew:
brew install kubeseal
As an alternative, download the kubeseal binary from GitHub.
2. Deploy the Sealed Secrets controller
Create a Flux Helm repository resource that points to the sealed-secrets Helm chart:
flux create source helm sealed-secrets \
--url https://bitnami-labs.github.io/sealed-secrets \
--interval 1h \
--export \
> sealed-secrets-source.yaml
Create a Flux Helm release resource to install the latest version:
flux create helmrelease sealed-secrets \
--interval=1h \
--release-name=sealed-secrets \
--target-namespace=flux-system \
--source=HelmRepository/sealed-secrets \
--chart=sealed-secrets \
--chart-version=">=1.15.0-0" \
--crds=CreateReplace \
--export \
> sealed-secrets-release.yaml
Deploy the Sealed Secrets controller via GitOps:
git add sealed-secrets-source.yaml sealed-secrets-release.yaml
git commit -m "Deploy Bitnami sealed secrets"
git push
Wait for the reconciliation process to notice the change and automatically deploy the controller.
3. Retrieve the public key
Retrieve the public key from the Sealed Secrets controller with kubeseal:
kubeseal --fetch-cert \
--controller-name=sealed-secrets \
--controller-namespace=flux-system \
> pub-sealed-secrets.pem
4. Create a SealedSecret
Create a standard Kubernetes Secret:
kubectl create secret generic my-secret \
--from-literal=secret=1234 \
--dry-run=client \
-o yaml > my-secret.yaml
Seal the Secret in a SealedSecret with kubeseal using the controller public key:
kubeseal --format=yaml --cert=pub-sealed-secrets.pem \
< my-secret.yaml \
> my-secret-sealed.yaml
5. Commit the SealedSecret
Discard the original Secret and commit the SealedSecret in the repository:
rm my-secret.yaml
git add my-secret-sealed.yaml
git commit -m "Add sealed my-secret"
git push
6. Wait for the Sealed Secret to be deployed
Wait for the SealedSecret to be deployed:
kubectl get SealedSecret -w
Retrieve the original Secret:
kubectl get secret my-secret -o jsonpath='{.data.secret}' | base64 -d -
Expected output:
1234
That is it ✌️
Conclusions
Bitnami is a great tool to keep Kubernetes Secrets secret. It is automation and GitOps friendly. However, its usage is confined to Kubernetes Secrets. In other words, it does not keep other non-Kubernetes Secrets (e.g. ssh keys, passwords, etc.) safe in my repository. For that, there are other solutions available, like Mozilla SOPS, which is also compatible with Flux.
I hope you enjoyed the article. If you would like to try a complete end-to-end example of Bitnami Sealed Secrets in Flux, you can continue here.
