Managing Kubernetes secrets with CyberArk Conjur

Jacopo Nardiello -

Conjur Secrets Manager is a solution from Cyberark to remove secrets from the source code and safely deliver them to your containerized applications.

As there is more than one way to cook an egg, there are different strategies to provide credentials to your application.

Having multiple options is great as it allows you to choose the solution that best suits your environment but sometimes too many possibilities can be confusing.

Knowing the pros and cons of every strategy can help you make the right decision.

Secrets delivery options

  1. Summon + InitContainer Authenticator
  2. Summon + Sidecar Authenticator container
  3. Api Call + Sidecar Authenticator container
  4. Secretless Broker
  5. Push secrets to Kubernetes/Openshift

Building Blocks

Summon


Summon is an Open Source utility developed by Cyberark. It provides an interface for reading a secrets.yml file, fetching secrets from a trusted store (Conjur Secrets Manager in our case) and exporting secret values to a sub-process environment (i.e. your application process).

Pros

  • Open source project by Cyberark itself.
  • No application code change is required except for the application configuration where the credentials are defined.

Cons

  • Seamless rotations are not supported - requires a pod restart when password changes.
  • Deployments require more steps: you need to add summon and the summon provider (summon-conjur in our case) to the Dockerfile of your application image as shown here.

Authenticator

Raison d'être for the Authenticator container is to fetch a short-lived token (Time To Live: 8 minutes) from the Conjur Follower and share it by means of a memory volume with the application container so that summon can use it to fetch secrets from the Conjur Follower. For an in-depth explanation of this flow see here.

Conjur Follower

Read only replica of the Conjur Master. Runs as a pod in the Kubernetes cluster. Can be horizontally scaled with ease. All pods that run in the Kubernetes cluster connect exclusively to the Conjur Follower to fetch secrets.

Strategies in depth

Summon + InitContainer Authenticator

Use case short lived jobs/cronjobs.

Pros only two rest calls are done to the conjur follower for the whole pod lifespan.

Cons if the application container is restarted an infinite crash loop is triggered.

InitContainers start and execute before other containers in the same Pod and exit when their job is done. Unlike other containers they are not re-executed on pod restart apart from edge cases.

The result of the above behaviour is the following: if the application container gets restarted for any reason (failed liveness probe, OOM kill or node reboot without drain) the Authenticator InitContainer will not be re-executed.
No Authenticator InitContainer means no valid token which in turn means that the rest api call to fetch secrets made by Summon will fail and no credentials will be injected in the application container resulting in an endless crash loop.

Summon + Sidecar Container Authenticator

Use case Long lived applications that don’t need to aggressively rotate secrets.

Pros the Authentication Token is refreshed every 6 minutes, which coupled with its TTL of 8 minutes ensures that the token is always valid.

Cons every Sidecar Container Authenticator executes a rest api call every 6 minutes, something that must be taken in consideration for large clusters by adjusting the number of Conjur Followers replicas.

Push secrets to Kubernetes/Openshift

Use case soft migration from secrets managed by Kubernetes to secrets managed by Conjur.

Pros uses a model that it’s already well known by Administrators and Developers with Kubernetes experience.

Cons the least secure among all strategies. See secrets are not secrets . By pushing the Conjur secrets to Kubernetes secrets you are duplicating the secret store (Conjur + Etcd) ultimately losing the benefit of having a centralized vault and audit.


Secretless Broker

Use case hands down the best strategy available if the service connector for your target exists.

Pros

  • No secrets delivered to the application, the most secure strategy.
  • No application source code changes required.
  • Supports rotations.

Cons

As of today the following service connectors have been implemented:

Api Call + Sidecar Authenticator Container

Use case implement custom logic to fetch and rotate secrets that is not covered by the other strategies.

Pros

Cons

  • Requires the application's source code to be modified.

Conclusions

After examining all the available options we can sketch a flowchart to help choose the best strategy for our needs: