Managing the Sealing Key

Sealed Secrets provide an effective means of managing secret information for your Kubernetes cluster, allowing you to check your secret information into a Version Control System (VCS) like Git. This allows you to manage your secrets via GitOps along with the rest of your cluster manifests.

However, the security of your system and its secrets depends on safely managing the sealing key. The sealing key is the key used by the Sealed Secret controller to decrypt your sealed secrets when they are applied to your cluster. As described in Managing Secrets, at cluster creation time the sealing key may be provided by you or automatically generated by wkp. In either case, it is the one piece of information that must be controlled. Anyone with access to the sealing key can decrypt any sealed secret that was encrypted with the corresponding certificate.

To ensure you can decrypt your sealed secrets in the event of a catastrophic cluster failure, save your sealing key in a secure location such as a password manager or a dedicated secret management system like Hashicorp Vault.

Key Renewal and Multiple Sealing Keys

By default, the Sealed Secrets controller periodically performs an operation called key renewal which creates a new sealing key and corresponding certificate (default period: 30 days). After a key renewal, any secrets sealed with kubeseal will be encrypted with the new certificate unless an older one is explicitly specified on the command line. The controller will then use the new sealing key to decrypt it.

If left alone, this will generate a growing set of sealing keys associated with the controller. The controller will use the appropriate key to decrypt each sealed secret depending on which certificate was used to create it. This means that any backup of sealing keys must take into account the new keys generated via key renewal. You have three options:

1. Turn off key renewal (not recommended)
2. Periodically add any new sealing keys to your secret backup storage
3. Periodically re-encrypt your existing sealed secrets with the current sealing key (recommended)

The third option is supported via kubeseal with the re-encrypt flag:

kubeseal --re-encrypt < yourSealedSecret.yaml > aTempFile.yaml && mv aTempFile.yaml yourSealedSecret.yaml

Once you perform this operation for each sealed secret, you are free to label any preexisting sealing key secret as compromised:

kubectl --namespace kube-system label secrets yourSealedSecretKey sealedsecrets.bitnami.com/sealed-secrets-key=compromised --overwrite=true

All the keys may be located via:

kubectl get secret --namespace kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

A restart of the sealed secret controller will ignore any compromised keys.

As discussed in Managing Secrets, key renewal (automatic or manual) is not a substitute for rotating your actual secrets. In particular, it's important to keep in mind that the security leak caused by a breached sealing key can not be corrected merely by removing and regenerating the sealing key. You need to assume that all secrets encrypted with the sealing key are compromised and rotate your secret information.

Recoverying sealing keys from the cluster

For option 2 above (or if you did not back up your keys), you can recover your keys as long as your cluster is operational. The following command will extract the keys from your cluster:

kubectl get secret --namespace kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key --output yaml > yourKeyFile.yaml

You can apply them to a cluster as part of disaster recovery via:

kubectl apply --filename yourKeyFile.yaml

If the controller is already running, restart the controller to install the recovered keys:

kubectl delete pod --namespace kube-system -l name=sealed-secrets-controller

Recoverying secrets after a cluster failure

If your cluster has failed and you have sealed secrets stored in version control or on your filesystem, you can recover your secret information provided you backed up your sealing keys as described above. You can decrypt your sealed secrets using:

kubeseal --recovery-unseal --recovery-private-key yourSealingKey1,yourSealingKey2,... < yourSealedSecret.yaml

or

kubeseal --recovery-unseal --recovery-private-key yourSealingKey1 --recovery-private-key yourSealingKey2 ... < yourSealedSecret.yaml