Seal the Secret !!
2022-Aug-18
Seal the Secret !!
By:DALEEP SINGH
2022-Aug-18 09:08:27
DevOps culture is being adapted everywhere, developers put their codes and share using source control management systems such as GitHub. This could very well include confidential information such as user-name, passwords, keys, etc, which is needed when the code goes for build and deployment. The very similar is with GitOps also where admins would be storing the platform, environment specific information in such SCM and there have been many incidents where confidential data gets leaked and environments compromised using the information available.
With more and more applications being migrated to cloud-native and platforms moving from bare metals, and virtual machines to containers, Kubernetes is the most popular platform used by developers and admins alike for environment provisioning and onboarding applications.
Containers being stateless, Kubernetes provides resources like ConfigMaps and Secrets to externalize the configuration needed by the applications. Using secret ensures that you don't include sensitive information in your application code and it could be provided as an independent resource. Developers would write a Secret resource definition that would contain the confidential data and is needed by the application. The secret is pushed to source control systems along with the application code. This could very well be code needed to provision the infrastructure as part of Infrastructure-As-Code and could contain, for example, user credentials to pull the container images from the enterprise repository.
Till this point, this all makes sense, keeping confidential information separate and in a secret resource, however, secrets are just base64 encoded and are non-encrypted. This is good enough to obscure the data, however, not good enough to prevent information from being decoded. The information can be decoded as easy as opening a child's piggy bank.
To demonstrate the installation and use of SealedSecret, I am using a Kubernetes cluster with 01 master and 02 worker nodes.
First lets see a secret resource definition which can easily be decoded. Imagine, storing this information on GitHub and someone getting it and decoding it in no time and then, compromising the application/cluster.
This is the encoded value which contains the registry information along with credentials to access the images on the registry. See how easily I am able to decode and get the values in clear text. ( The actual values are blurred for obvious reasons ;-) ).
Then what is the solution?
The solution is to encrypt the Secret resource into a SealedSecret, which can then be stored on source control systems. The SealedSecret resource can be decrypted only by the controller running on the Kubernetes cluster and no one else. This makes storing your Secrets safe.
The SealedSecrets consists of two parts:
- A cluster-side controller/operator that runs on the Kubernetes cluster
- A client-side utility called kubeseal is used to encrypt a regular Kubernetes Secret object into a SealedSecret. The kubeseal utility uses asymmetric crypto to encrypt secrets that only the controller can decrypt. Once decrypted by the controller, the enclosed Secret can be used exactly like a regular Kubernetes Secret.
The SealedSecret project is available on gitHub at https://github.com/bitnami-labs/sealed-secrets/releases. We can download the controller and Kubeseal utility from here.
Execute below mentioned command to deploy SealedSecret controller.
#kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.1/controller.yaml
Once you deploy the manifest it will create the SealedSecret resource and install the controller into kube-system namespace, create a service account and other required RBAC roles. The controller will start, generate a key pair, and will be ready for operation. The controller generates its own certificates when is deployed for the first time, it also manages the renewal for you. But you can also bring your own certificates so the controller can consume them as well. (https://github.com/bitnami-labs/sealed-secrets/blob/main/docs/bring-your-own-certificates.md).
With SealedSecret, we pass the secret resource definition to kubeseal utility which encrypts the secret and create a new SealedSecret type of resource which can be stored in source control repositories and is completely secure.
I have created a secret defintion yaml file ( shown below) which is passed as input to kubeseal utility. This utility will pass the output to STDOUT, however, I have redirected it to a file for storing to source control system.
On inpecting the db-creds-sealed-secret.yaml, it shows the resource type as SealedSecret and the values are now under encryptedData field.
As of now, we don't have the Kubernetes secret resource created. Once we pass SealedSecret definition to kubectl, it will also create a Kubernetes secret resource.
We need to understand very carefully that once the sealedSecret resource is created in kubernetes, secret resource is automatically created and someone with access to the cluster can decode the secret resource very easily. SealedSecret will not be able to help us in that case, hence, taking care of proper RBAC policies and access control should be applied for access to secret resources.
Deleting the SealedSecret resource will also remove the Kubernetes secret resource as well.
Of course, kubernetes allows us to configure encryption of secret data at rest, ie, when it is stored in ETCD. Documentation is available at kubernetes docs (https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/).
Now that we understand how we can store sensitive information on source control systems and also can encrypt the secrets at rest in ETCD, I am sure we will be able to protect the information better.
