Skip to content

External Secrets and Immutable Target

If you have worked with External Secrets Operator before, then you know how it eases the operation of managing the secrets in the Kubernetes cluster. It supports many backends and is very powerful.

However, there is a nuance. The External Secrets Operator allows you to define an immutable target secret, sealing the secret shut from future changes unless explicitly deleted and recreated, which is perfect if you never want to modify the secret. But, change is the only constant in the world of IT, and you might want to change the secret in the future. This is where immutable can catch you off guard, as it did mine. This is my story and how I solved it.

Introduction

External Secrets Operator has support for a variety of secret backends, including AWS Secrets Manager, Azure Key Vault, Google Secret Manager, HashiCorp Vault, etc. That said, if you have different backends for your secret management, this is the perfect solution for you.

With External Secrets, you can define one or more SecretStore or ClusterSecretStore to read the secrets from a specific backend and create the Kubernetes Secret resource with the name and namespace you specify. This is a great way to store secrets in a secure and encrypted location, e.g., AWS Parameter Store, and granting the External Secret Operator the permission to fetch (and if desired create/update secrets) in the backend.

A Working Example

First things first, let's spin up a Kubernetes cluster. I am using kind1 to create a local and lightweight Kubernetes cluster.

Bash
kind create cluster --image=kindest/node:v1.29.2

On Versioning

It's always a good idea to pin your dependencies to a specific version. I would go as far as to say that you should do it even in your personal projects. You will ensure reproducibility and avoid surprises for others and for your future self.

After this command, I will see a single-node cluster on my machine.

Install the External Secrets Operator2

I prefer the Helm installation method since it is deterministic and behaves as expected when pinning to a specific version.

Bash
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets --version=0.9.x

Create a Secret Store

A Secret Store (or Cluster Secret Store) is the resource talking to your encrypted secrets management backend. Here's an example of how to create a Cluster Secret Store with the backend of AWS Parameter Store.

css.yml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-parameter-store
spec:
  provider:
    aws:
      auth:
        secretRef:
          accessKeyIDSecretRef:
            key: AWS_ACCESS_KEY_ID
            name: aws-ssm-user
            namespace: default
          secretAccessKeySecretRef:
            key: AWS_SECRET_ACCESS_KEY
            name: aws-ssm-user
            namespace: default
      region: eu-central-1
      role: arn:aws:iam::123456789012:role/aws-ssm-user
      service: ParameterStore

AWS IAM User

Be mindful that this secret store will need a way to access the secrets in your backend. In our example, that means we have to create an IAM User with access key and secret access key, and pass the credentials as Kubernetes Secret with the name aws-ssm-user (the highlighting lines in the above snippet).

To do that, we'll get help from our good friend OpenTofu.

variables.tf
variable "username" {
  default = "my-user"
  type    = string
}

variable "secret_name" {
  default = "aws-ssm-user"
  type    = string
}

variable "secret_namespace" {
  default = "default"
  type    = string
}

variable "external_secret_name" {
  default = "aws-parameter-store"
  type    = string
}
iam.tf
data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "aws_iam_user" "this" {
  name = var.username
}

data "aws_iam_policy_document" "assume_role" {
  statement {
    actions   = ["sts:AssumeRole"]
    effect    = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_user.this.arn]
    }
  }
}

resource "aws_iam_role" "this" {
  name               = "${var.username}-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

data "aws_iam_policy_document" "ssm_readonly" {
  statement {
    actions = [
      "ssm:GetParameter",
      "ssm:GetParameters",
      "ssm:GetParametersByPath",
    ]
    effect    = "Allow"
    resources = ["*"]
  }
}

resource "aws_iam_policy" "this" {
  name        = "${var.username}-policy"
  description = "Policy for readonly access to AWS Parameter Store"

  policy = data.aws_iam_policy_document.ssm_readonly.json
}

resource "aws_iam_role_policy_attachment" "this" {
  role       = aws_iam_role.this.name
  policy_arn = aws_iam_policy.this.arn
}

resource "aws_iam_access_key" "this" {
  user = aws_iam_user.this.name
}

Info

All the examples in this post are tested and working. To reproduce it, simply run tofu init and tofu apply.

So far, so good. There is only one issue! The Cluster Secret Store is not a templated resource and as such, except for the secret keys AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, we would have to specify both region and the role manually as hard-coded values in the YAML manifest.

That is not ideal, not by a long shot.

That's why, we'll use the same powerful HCL syntax we've been using so far to create the Cluster Secret Store from inside the OpenTofu files.

Create the Cluster Secret Store

kubernetes.tf
resource "kubernetes_secret" "this" {
  metadata {
    name      = var.secret_name
    namespace = var.secret_namespace
  }

  binary_data = {
    AWS_ACCESS_KEY_ID     = base64encode(aws_iam_access_key.this.id)
    AWS_SECRET_ACCESS_KEY = base64encode(aws_iam_access_key.this.secret)
  }

  type = "Opaque"
}

resource "kubernetes_manifest" "this" {
  manifest = {
    apiVersion = "external-secrets.io/v1beta1"
    kind       = "ClusterSecretStore"

    metadata = {
      name = var.external_secret_name
    }

    spec = {
      provider = {
        aws = {
          region  = data.aws_region.current.name
          role    = aws_iam_role.this.arn
          service = "ParameterStore"

          auth = {
            secretRef = {
              accessKeyIDSecretRef = {
                key       = "AWS_ACCESS_KEY_ID"
                name      = var.secret_name
                namespace = var.secret_namespace
              }
              secretAccessKeySecretRef = {
                key       = "AWS_SECRET_ACCESS_KEY"
                name      = var.secret_name
                namespace = var.secret_namespace
              }
            }
          }
        }
      }
    }
  }
}

You will notice, promptly, that we no longer have to write any hard-coded values as we had to do earlier in the css.yml example in our definitions and everything is being initialized from the resources being created in the same stack.

That's a powerful approach when provisioning and maintaining your infrastructure as it gives you, at its minimum, a reproducible and consistent environment where you can use for your day-to-day operations as well as your disaster recovery plans.

Reference a Secret Stored in AWS SSM

Now, onwards we go. The next task is to create a secret in AWS SSM and use it inside our cluster by the way of External Secrets Operator.

The task is simple, but we'll keep the tradition alive by using OpenTofu to create an encrypted secret in AWS SSM.

Let's imagine we want to store some fake password in our secret store.

ssm.tf
variable "mongo_root_password" {
  type      = string
  sensitive = true
  default   = "ThisIsNotASecurePassword"
}

resource "aws_ssm_parameter" "this" {
  name  = "/prod/mongodb-atlas/passwords/root"
  type  = "SecureString"
  value = var.mongo_root_password
}

Reference the Secret From Inside the Cluster

Now, the moment of truth. This is where it all comes together. We'll create the External Secret resource to reference the secret we created in the previous step.

external-secret.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app
  namespace: default
spec:
  data:
    - remoteRef:
        key: /prod/mongodb-atlas/passwords/root
      secretKey: MONGO_ROOT_PASSWORD
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-parameter-store
  target:
    immutable: false

To create this resource, simply use kubectl:

Bash
kubectl apply -f external-secret.yml

Notice the highlighted lines as it is the topic of this post.

Is the Secre Correct?

To realize if the secret is initialized correctly, we can investigate the Kubernetes Secret resource:

Bash
$ kubectl get secret my-app -o jsonpath='{.data.MONGO_ROOT_PASSWORD}' | base64 --decode && echo
ThisIsNotASecurePassword

And lo and behold, the secret is there and it is correct.

Tip

The use of echo at the end of the last command is not mandatory, however, it will clear the line after the password is printed and your shell prompt will not stay on the same line as the password.

Let's Change the Password Now

Now, this is all good and sexy. But what happens if we wanted to rotate our password, i.e., change it in the backend (AWS SSM in this case). The expected behavior is that the External Secret Operator would pick up the change as frequent as its refreshInterval and update the secret in the cluster.

However, as mentioned at the beginning of this post, immutable plays a huge role in this expectation as we shall see shortly.

Let's modify the password using the OpenTofu again.

Bash
tofu apply -var mongo_root_password=SomethingDifferent

We will have to wait for the maximum of refreshInterval (1m in this case) for the External Secret Operator to pick up the change and update the secret.

After that, another look at the secret will reveal that the password.

Bash
$ kubectl get secret my-app \
  -o jsonpath='{.data.MONGO_ROOT_PASSWORD}' | \
  base64 --decode && echo
SomethingDifferent

It indeed change the password.

Let's Break it a Little

Now, we have seen that the External Secret is behaving as expected. But, let's change the immutable field in its spec and observe the behavior.

external-secret-v2.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app
  namespace: default
spec:
  data:
    - remoteRef:
        key: /prod/mongodb-atlas/passwords/root
      secretKey: MONGO_ROOT_PASSWORD
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-parameter-store
  target:
    immutable: true

Updating the External Secret will not prove our point here! But, if we remove and recreat the External Secret, we won't embarass ourselves by opening such topic.

Bash
kubectl delete -f external-secret.yml
kubectl apply -f external-secret-v2.yml

Now, as we can see from the target Secret, the immutable flag is set to true.

Bash
$ kubectl get secret my-app -o jsonpath='{.immutable}' && echo
true

You should have noticed that we haven't changed the second version of the secret and it should still be SomethingDifferent.

This is the point and we can even make sure by looking into the secret again.

Bash
$ kubectl get secret my-app \
  -o jsonpath='{.data.MONGO_ROOT_PASSWORD}' | \
  base64 --decode && echo
SomethingDifferent

But, what if we want to update the secret again?

Bash
tofu apply -var mongo_root_password=AnotherPassword

Even after waiting for as long as more than the refreshInterval, the secret will not change.

Bash
$ kubectl get secret my-app \
  -o jsonpath='{.data.MONGO_ROOT_PASSWORD}' | \
  base64 --decode && echo
SomethingDifferent

But, wait. Didn't we just change the password? Shall we check the AWS SSM?

Bash
$ aws ssm get-parameters \
  --names /prod/mongodb-atlas/passwords/root \
  --with-decryption \
  --query Parameters[0].Value --output text
AnotherPassword

Wow. 🤯

This is not what we expected. The app will surely break.

And so, that is the whole point of writing this long post. The immutable flag is a double-edged sword. It is a great feature if you want to make sure that the secret is not changed, but it is a disaster if you want to change the secret in the future.

In other words, secret rotation, which is a industry security best practice, is not possible with the immutable flag set to true.

The alternative is of course to set it to false, which is even a worse idea in my opinion, since if there isn't a proper policy or RBAC in place, anyone with access to the cluster can, accidentally or otherwise, change the content of the secret.

Conclusion

The External Secrets Operator is a powerful tool to manage secrets in the Kubernetes cluster. It supports many backends and is very flexible. However, the immutable flag is not at its best behavior and you should be extra cautious when using it in your production environment.

Thanks for reading and I hope you enjoyed this post. If you have any questions or comments, please feel free to reach out.

Happy hacking!

If you enjoy these contents, please consider consider supporting the blog by the way of GitHub Sponsors 💓.

Versions

To help with reproducibility, I will include the versions of the providers in this post.

versions.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.37"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.26"
    }
  }

  required_version = "< 2"
}

Source Code

The code for this post is available from the following link.

Source code

  1. https://github.com/kubernetes-sigs/kind ↩

  2. https://external-secrets.io/latest/introduction/getting-started/ ↩