Cloud-Native Secret Management: OIDC in K8s Explained¶

External Secrets is the de-facto choice for secrets management in Kubernetes clusters. It simplifies the task of the administrator(s) of the cluster, ensuring only the secrets that are explicitly defined are present and accessible.

It comes with many great features but most important than all is its integration with major cloud providers.

In this blog post you will learn how to deploy it without hard-coded credentials and using only the power of OpenID Connect for trust relationship between services.

Introduction¶

I like External Secrets Operator a lot. I rarely second guess my choice deploying it in any Kubernetes cluster I provision and/or manage.

It enhances the security posture of my workload, while ensuring the right APIs are called and the desired secrets are available to my pods at the right namespace.

The task of secret decryption is also handled by the cloud provider, which removes the heavy lifting of key management and rotations!

On the other hand, OpenID Connect is the authentication protocol that allows for interactions and communications between services that support it.

The main idea is that by establishing a trust relationship between the identity provider and the service provider, you will allow the the entities of the former to access and talk to the latter.

The objective for today's blog post is to get the best of both of these ideas and deploy External Secrets Operator in major cloud providers.

We will discuss managed Kubernetes clusters as well as self-managed ones.

By the end of this blog post, you should have a good idea of how to do this in AWS, Azure, GCP, in addition to bare-metal Kubernetes deployments as a final bonus.

What is OpenID Connect?¶

The first question we gotta answer for those of you who are not familiar is the OIDC protocol.

The idea is pretty simple actually; we want the entities/identities from one service (the Identity Provider) to be able to talk to the other (the Service Provider).

The way we do that is in two steps:

1. We first establish a trust relationship between the Identity Provider and the Service Provider. What this does is to let the service provider know that the folks from the identity provider are to be authenticated to the system and can make API calls without receiving 401.
2. Once the trust relationship is setup, we can grant the required permissions and RBAC to the target identity coming from the identity provider & let it make the API call(s). An example can be aws s3 ls s3://my-example-bucket running from inside the pods of an AWS EKS; unless the required permissions are granted beforehand, the result will be a 403 unauthorized!

Here's a summary of this interaction.

sequenceDiagram
    actor Identity
    participant IDP as Identity Provider
    participant SP as Service Provider

    IDP->>SP: Establish Trust Relationship
    Note right of SP: Trust relationship established

    Identity->>SP: Authenticate
    alt Trust Established
        SP->>Identity: Access Granted
    else No Trust Established
        SP->>Identity: 401 Unauthorized
    end

Now let's get on with implementing it in different vendors.

Amazon Web Services (AWS)¶

The AWS EKS has many integrations with the current AWS services.

They actually have a habit of building their products with native support for their other products.

They have a really good reason to do so because that will make you buy more and pay more.

But enough complaining, let's get serious.

Registering EKS OpenID Connect Provider¶

The first order of business is to ensure that the pods in your AWS EKS is able to talk to other services and authenticate successfully.

That comes in the form of registering the Identity Provider of your cluster to the AWS IAM¹.

You can achieve this either from your OpenTofu/Terraform code, or with a simple CLI command.

eksctl utils associate-iam-oidc-provider --cluster <my-cluster> --approve

This will give us something in the following format:

oidc.eks.REGION.amazonaws.com/id/OIDC_PROVIDER_ID

We will use it shortly when creating IAM Policy.

That will give you the first step ; establishing a trust relationship between the two services.

Create the AWS IAM Policy¶

The next step is to grant the pods of External Secrets access to read the secrets from our AWS Secrets Manager, as well as to decrypt the secure secrets; we are actually giving it access to Parameter Store, which provides a limited free access to store and access secure values².

{
  "Statement": [
    {
      "Action": [
        "ssm:GetParameter",
        "ssm:GetParameters",
        "ssm:GetParametersByPath"
      ],
      "Condition": {
        "StringEquals": {
          "oidc.eks.REGION.amazonaws.com/id/OIDC_PROVIDER_ID:aud": "sts.amazonaws.com",
          "oidc.eks.REGION.amazonaws.com/id/OIDC_PROVIDER_ID:sub": "system:serviceaccount:external-secrets:external-secrets"
        }
      },
      "Effect": "Allow",
      "Resource": [
        "arn:aws:ssm:REGION:ACCOUNT_ID:parameter/*"
      ]
    },
    {
      "Action": [
        "ssm:DescribeParameters"
      ],
      "Condition": {
        "StringEquals": {
          "oidc.eks.REGION.amazonaws.com/id/OIDC_PROVIDER_ID:aud": "sts.amazonaws.com",
          "oidc.eks.REGION.amazonaws.com/id/OIDC_PROVIDER_ID:sub": "system:serviceaccount:external-secrets:external-secrets"
        }
      },
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Condition": {
        "StringEquals": {
          "oidc.eks.REGION.amazonaws.com/id/OIDC_PROVIDER_ID:aud": "sts.amazonaws.com",
          "oidc.eks.REGION.amazonaws.com/id/OIDC_PROVIDER_ID:sub": "system:serviceaccount:external-secrets:external-secrets"
        }
      },
      "Effect": "Allow",
      "Resource": [
        "arn:aws:kms:REGION:ACCOUNT_ID:key/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

aws iam create-policy \
  --policy-name external-secrets \
  --policy-document file://aws-iam-policy.json

And the obvious next step is to allow our pod to have access to these permissions through short-lived credentials³.

The following commands create two resources:

1. AWS IAM Role.
2. A Kubernetes ServiceAccount.

eksctl create iamserviceaccount \
  --name external-secrets \
  --namespace external-secrets \
  --cluster <my-cluster> \
  --role-name external-secrets \
  --attach-policy-arn arn:aws:iam::111122223333:policy/external-secrets \
  --approve

This command will result in creating an AWS IAM Role similar to the following:

{
  "Statement": [
    {
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "OIDC_PROVIDER_ID:aud": "sts.amazonaws.com",
          "OIDC_PROVIDER_ID:sub": "system:serviceaccount:external-secrets:external-secrets"
        }
      },
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/OIDC_PROVIDER_ID"
      }
    }
  ],
  "Version": "2012-10-17"
}

Once we have this, we're ready to deploy the External Secrets Operator.

NOTE: You will need the Amazon Identity Webhook operator installed in your cluster, which comes pre-installed in AWS EKS⁴.

Deploy ESO Helm Chart in AWS AKS¶

We would need both the CRD installation as well as the corresponding ServiceAccount annotation for the IAM trust relationship to work:

---
installCRDs: true
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/external-secrets
  create: false

And now we can install it:

helm install -n external-secrets \
  external-secrets \
  external-secrets/external-secrets \
  --values values.yml \
  --version 0.15.x

Let's finalize the AWS with the ClusterSecretStore creation⁵.

And to verify that it works:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-parameter-store
spec:
  provider:
    aws:
      region: eu-central-1
      service: ParameterStore

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: hello
  namespace: default
spec:
  data:
    - remoteRef:
        key: /path/to/key
      secretKey: SOME_ENV_VAR
  refreshInterval: 24h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-parameter-store

That wraps up the AWS part. Let's move on to Azure.

Microsoft Azure Cloud¶

With Azure AKS, you need the Workload Identity Operator installed in your cluster⁶.

Fortunately, it comes pre-installed if you're using a managed AKS cluster.

The idea is quite about the same as before; we aim to establish a trust relationship from the Identity Provider (the Kubernetes cluster) to the Service Provider (the Azure API) so that the pods from our identity provider can issue API calls and not get 401 error.

Establish Trust Relationship Between AKS and Azure API¶

To achieve this, we use our good friend OpenTofu this time.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "< 5"
    }
  }

  required_version = "< 2"
}

variable "key_vault_id" {
  type     = string
  nullable = false
}

data "azurerm_subscription" "current" {}

data "azurerm_kubernetes_cluster" "this" {
  name                = "my-aks-cluster"
  resource_group_name = "my-resource-group"
}

resource "azurerm_user_assigned_identity" "this" {
  name                = "vault-external-secrets"
  resource_group_name = "my-resource-group"
  location            = "Germany West Central"
}

resource "azurerm_federated_identity_credential" "this" {
  name                = "external-secrets"
  resource_group_name = "my-resource-group"
  audience            = ["api://AzureADTokenExchange"]
  issuer              = data.azurerm_kubernetes_cluster.this.oidc_issuer_url
  parent_id           = azurerm_user_assigned_identity.this.id
  subject             = "system:serviceaccount:external-secrets:external-secrets"
}

resource "azurerm_key_vault_access_policy" "this" {
  key_vault_id = var.key_vault_id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = azurerm_user_assigned_identity.this.principal_id

  secret_permissions = [
    "Get",
  ]

  certificate_permissions = [
    "Get",
  ]
}

output "tenant_id" {
  value = data.azurerm_subscription.current.tenant_id
}

output "client_id" {
  value = azurerm_user_assigned_identity.this.client_id
}

This is only half the battle, although the most important part.

Deploy ESO Helm Chart in Azure AKS¶

We'd first need the required label set on the target pod, as well as the right annotations set on the ServiceAccount.

---
installCRDs: true
podLabels:
  azure.workload.identity/use: "true"
serviceAccount:
  annotations:
    azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
    azure.workload.identity/tenant-id: 00000000-0000-0000-0000-000000000000

And the installation is just as before using Helm.

The ClusterSecretStore will look like the following.

Let's verify the setup:

---
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: azure-keyvault
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: https://my-keyvault.vault.azure.net/

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: hello
  namespace: default
spec:
  data:
    - remoteRef:
        key: path-to-key
      secretKey: SOME_ENV_VAR
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: azure-keyvault

Two down, one to go.

GCP Cloud¶

This is the last major cloud provider we'll cover in this blog post.

You will need the Workload Identity enabled on your GKE cluster for this method to work properly.

gcloud container clusters create [CLUSTER_NAME] \
  --workload-pool=[PROJECT_ID].svc.id.goog

Let's first create the required resources that will allow Workload Federation within our GCP account⁸.

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "< 7"
    }
  }

  required_version = "< 2"
}

data "google_project" "current" {}

resource "google_project_iam_member" "external_secrets" {
  project = data.google_project.current.project_id
  role    = "roles/secretmanager.secretAccessor"

  member = format(
    "principal://iam.googleapis.com/projects/%s/locations/global/workloadIdentityPools/%s.svc.id.goog/subject/ns/%s/sa/%s",
    data.google_project.current.number,
    data.google_project.current.project_id,
    "external-secrets",
    "external-secrets",
  )
}

Fortunately, you wouldn't need to annotate your Service Account this time because that is take care of under the hood. You just gotta make sure that the Servic Account is mounted into the pod.

---
installCRDs: true
serviceAccount:
  automount: true

And so business as usual, we'll install this using the Helm CLI.

Let's verify the setup:

---
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-manager
spec:
  provider:
    gcpsm:
      projectID: my-gcp-project

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: hello
  namespace: default
spec:
  data:
    - remoteRef:
        key: path-to-key
      secretKey: SOME_ENV_VAR
  refreshInterval: 24h
  secretStoreRef:
    kind: ClusterSecretStore
    name: gcp-secret-manager

Bonus: Self-Managed Kubernetes Clusters¶

Now, this all sounds so very good.

In fact, cloud providers take care of a lot of other things for the above methods to work properly.

But, what if you were running a self-managed Kubernetes cluster?

How would you then allow the pods (identities) of your cluster to access the API of other Service Providers?

How would you allow the External Secrets Operator from a self-managed Kubernetes cluster to read secrets from AWS Parameter Store and create the corresponding Secret in the correct namespace?

The answer lies in the OpenID Connect.

Based on RFC 8414⁹, the systems that provide the /.well-known/openid-configuration can be considered OIDC-compliant.

The output of this endpoint will be the details on how your Identity Provider signs the tokens for its identity and the URL for public key that can be used to verify those tokens.

You can try this command on any Kubernetes cluster right now to see for yourself.

kubectl get --raw /.well-known/openid-configuration

The result will look something like the following:

---
id_token_signing_alg_values_supported:
  - RS256
issuer: https://example.com
jwks_uri: https://example.com/openid/v1/jwks
response_types_supported:
  - id_token
subject_types_supported:
  - public

Now, to answer the first few questions of this section...

If you copy the contents of the following two endpoints from your Kubernetes cluster...

1. /.well-known/openid-configuration
2. /openid/v1/jwks

... and store them somewhere that can be accessed behind secure HTTPS (signed by trusted CA) and serve them as static files, then any Service Provider can establish a trust relationship with your cluster and allow the pods within the cluster to make those beautiful API calls.

That's the whole magic really!

Try it with any Kubernetes cluster and it just works. You only need to serve the contents of those two endpoints to the public internet.

Isn't that exciting!? Doesn't that excite you even a little bit?

It's all so good. So good to be true. But it is. It just is.

Conclusion¶

I am blown away by how far away we've come from the early days of internet down to the days that we can simply authenticate the entities of one system to the other without passing any access-key and secret-key anywhere.

Not passing secrets means no secret rotation, no key management, no overhead, no chores, no toil, and just pure bliss!

I love the world we live in because there's OpenID Connect in it!

As you've seen throughout this blog post, some cloud providers call it by other names but that's just a marketing term in my humble opinion.

The fact stands still, that the OpenID Connect is one of the most remarkable inventions of mankind to this day, at least in my opinion.

Until next time , ciao & happy coding!

Subscribe to Newsletter Subscribe to RSS Feed

Share on Share on Share on Share on

Comments