Skip to content

How to Setup Preview Environments with FluxCD in Kubernetes

Preview environment is where you see a live state of your changes from your pull request before being merged into the default branch. It gives you a look'n feel of what it would be like if you merged your changes.

Kubernetes on the other hand, is what powers the production setups. But that's not all it can do for you. I have spun up preview environments in Kubernetes with different technologies in the past.

And in this blog post, I will show you how to achive this using FluxCD Operator.

Introduction

For most of the frontend or full-stack applications, you rarely second guess using anything but some of the well-known PaaS providers platforms like Vercel, Netlify, etc.

Not only they make it very easy to integrate your Git provider and deploy your application with the least friction, but also you will benefit from out-of-the-box preview deployment with little to no configuration.

However, those platforms have two downsides (from my point of view):

  1. They are generalized, meaning that some of the specifics of your application may or may not be compatible with their offering.
  2. Most importantly, they are mainly focused on JavaScript ecosystem, whether that be NodeJS, NextJS, Svelte, etc. When it comes to backend application deployments such as Golang or Python, they fall short!

On the flip side, it's not unheard of for folks to ditch Kubernetes altogether and go full in-house with a customized version of what they seem fit for their development environments1.

They claim, and I quote:

[...] we’ve found that Kubernetes is not the right choice for building development environments.

And not shortly after:

This is not a story of whether or not to use Kubernetes for production workloads.

So, in a nutsheel, they are at a point where Kubernetes does not work for them.

No complaint here. Pick what works best for you!

Why Kubernetes?

However, me personally, I rarely leave or second guess Kubernetes clusters.

Even for the obvious challenges of keeping storage accessible when the pod moves around between nodes, I have personally found RWX storage classes to be a good solution, e.g., Ceph, NFS, etc.

The reasons below are not exhaustive, but they provide my take on why I believe Kubernetes is a superior choice for not just prod, but also anything in between:

  1. Deployment frequency: matched with GitOps, there's no better way to deploy your applications as fast as possible2. ⏩
  2. Mean time to recovery: If you have everything in Git, you only need a backup of your data; your cluster is always backed up in VCS. 🕐
  3. Zero downtime rollouts: What does it take for a supervisord or docker-compose to deliver the same!?
  4. Preview deployments: This will be the main story of this blog post! ⚡
  5. Engineering overhead/technical debt/maintainability: There can be books written on this topic, but in short, Kubernetes is a lot less overhead than, say, Ansible in GitHub Actions (I have done both BTW).
  6. GitOps (disaster recovery/drift detection): We touched on this earlier (point 1 & 2).
  7. Automated releases: Sit back and enjoy deployments with no manual intervention. What can be done by machines should be done by machines! 😎
  8. Self healing deployments: I'm sure every other tool will try as hard but Kubernetes is the best at this. 💪
  9. Monitoring & observability: What's the alternative? Deploy Prometheus systemd service on every machine? No sirree, not for me! 🏃‍♂️

What are we doing here?

When I read other people's blog posts, I skip straight to the part with the source code snippet. I'm not one for the theories!

I will spare you the same. Let's get to the point!

Here are the spec we want to address in this blog post.

  1. We want to deploy a Golang application to Kubernetes cluster.
  2. We want every pull request on the repository of our application to spin up a preview environment.
  3. Every preview environment should have a dedicated internet-accessible URL.
  4. The URL of the preview environment has to be commmented on the GitHub pull request.

Let's start to address each of these and hope that we won't have to break this blog post into multiple. 😅

Pre-requisites

  • A Kubernetes cluster. We're working with v1.32.
  • FluxCD installed on the cluster3. Currently v2.5.1.
  • GitHub repository for the application.
  • Helm CLI installed4. Currently v3.17.1.

Directory Structure

As per the tradition of our blog posts, here is the directory structure we'll be working on.

.
├── 10-app/
│   └── kustomize/
│       ├── base/
│       └── overlays/
│           └── preview/
└── 20-infra/
    ├── 30-flux-rbac/
    └── 40-preview-environment/

A Minimal Golang Application

Let's create our initial boilerplate:

cd 10-app/
go mod init fluxy_dummy
go get -u github.com/gin-gonic/gin

And the only file we need:

10-app/main.go
package main

import (
    "net/http"

    "github.com/gin-gonic/gin"
)

func main() {
    r := gin.Default()

    r.GET("/ping", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "message": "pong",
        })
    })

    r.Run()
}

We have our application ready. Commit and push this to its own repository and we're ready to create a base Kustomization stack for its deployment.

Kustomization vs. Helm

I am more in favor of Kustomization for its expressiveness and simplicity.

Templating languages do not scare me really; I even use Jinja2 templates in Ansible extensively.

But unless I am providing a generalized deployment stack for the community, I see no point in making my manifets more complex.

Kustomization is just that; simple and easy to understand. What you see is exactly what you get. No fuss, no templating, no nothing!

Kustomize the Deployment

We will now create the YAML manifets that will be used by the FluxCD Kustomization to deploy our application.

10-app/kustomize/base/serviceaccount.yml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluxy-dummy
automountServiceAccountToken: false
10-app/kustomize/base/deployment.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluxy-dummy
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    spec:
      containers:
        - image: ghcr.io/developer-friendly/fluxy-dummy
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /ping
              port: http
            initialDelaySeconds: 3
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
          name: fluxy-dummy
          ports:
            - containerPort: 8080
              name: http
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /ping
              port: http
            initialDelaySeconds: 3
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
            runAsGroup: 65534
            runAsNonRoot: true
            runAsUser: 65534
      initContainers: []
      securityContext:
        fsGroup: 65534
        fsGroupChangePolicy: Always
        seccompProfile:
          type: RuntimeDefault
        supplementalGroups: []
        sysctls: []
      serviceAccountName: fluxy-dummy
      terminationGracePeriodSeconds: 10
10-app/kustomize/base/service.yml
apiVersion: v1
kind: Service
metadata:
  name: fluxy-dummy
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
  type: ClusterIP
10-app/kustomize/base/kustomization.yml
resources:
  - serviceaccount.yml
  - service.yml
  - deployment.yml

images:
  - name: ghcr.io/developer-friendly/fluxy-dummy

labels:
  - includeSelectors: true
    includeTemplates: true
    pairs:
      app.kubernetes.io/component: fluxy-dummy
      app.kubernetes.io/instance: fluxy-dummy
      app.kubernetes.io/managed-by: Kustomize
      app.kubernetes.io/name: fluxy-dummy
      app.kubernetes.io/part-of: fluxy-dummy
      app.kubernetes.io/version: v1.0.0

namespace: default

And its overlay:

10-app/kustomize/overlays/preview/kustomization.yml
resources:
  - ../../base

labels:
  - includeSelectors: true
    pairs:
      preview: ${PR_NUMBER}

Notice the ${PR_NUMBER} we use in the overlay. This will be populated by the FluxCD postBuild configuration5.

This will be crucial when creating multiple Kubernetes Services with the same set of labels in the same namespace; the includeSelectors will ensure that the Service created for each stack is pointing to its corresponding Pod6.

FluxCD Operator

So far so good, but we've done nothing more than the usual application deployment.

I would say, that whatever that's been covered so far is a base requirement for any production setup.

We now move on to our main objective: preview deployments.

First things first, we gotta deploy the corresponding operator and its CRDs.

helm install -n flux-system \
  --create-namespace \
  flux-operator \
  oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator \
  --version=0.17.0

This will be the latest version of the FluxCD operator (as of this writing).

Prerequisite Kubernetes Resources

As per the official documentation7, we better create a dedicated namespace and a role binding to make sure no elevated permissions are given to the operator when deploying the preview environments.

20-infra/30-flux-rbac/namespace.yml
---
apiVersion: v1
kind: Namespace
metadata:
  name: staging
20-infra/30-flux-rbac/serviceaccount.yml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flux
  namespace: staging
20-infra/30-flux-rbac/rolebinding.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: flux
  namespace: staging
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
subjects:
  - kind: ServiceAccount
    name: flux
    namespace: staging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gateway-admin
rules:
  - apiGroups:
      - gateway.networking.k8s.io
    resources:
      - httproutes
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: flux-gateway-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gateway-admin
subjects:
  - kind: ServiceAccount
    name: flux
    namespace: staging

Preview Environment YAML Manifests

We're ready to leverage all our existing setup and create a preview environment for each of our pull requests.

20-infra/40-preview-environment/rsip.yml
---
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
  annotations:
    fluxcd.controlplane.io/reconcileEvery: 10s
  name: preview-deployments
  namespace: staging
spec:
  filter:
    labels:
      - preview
  secretRef:
    name: github-auth
  type: GitHubPullRequest
  url: https://github.com/developer-friendly/fluxy-dummy

To create the GitHub authentication token, and until FluxCD operator provides native support for GitHub Deploy Keys8, we're left with no other option but the infamous GitHub PAT.

flux -n staging create secret git github-auth \
  --url=https://github.com/developer-friendly/fluxy-dummy \
  --username=meysam81 \
  --password=${GITHUB_TOKEN}

Beware that we set the fluxcd.controlplane.io/reconcileEvery to 10s just for simplicity. In real-world scenarios you'd have a realistic number such as 5m or higher and set up the GitHub webhook to trigger the reconciliation9.

The following CRD is our main resource. It takes care of receiving the GitHub pull requests from the parent ResourceSetInputProvider and creating the specified resources.

20-infra/40-preview-environment/rset.yml
---
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: fluxy-dummy
  namespace: staging
  annotations:
    fluxcd.controlplane.io/reconcile: "enabled"
    fluxcd.controlplane.io/reconcileEvery: "10s"
    fluxcd.controlplane.io/reconcileTimeout: "1m"
spec:
  serviceAccountName: flux
  inputsFrom:
    - apiVersion: fluxcd.controlplane.io/v1
      kind: ResourceSetInputProvider
      name: preview-deployments
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      metadata:
        name: fluxy-dummy-pr<< inputs.id >>
        namespace: staging
      spec:
        interval: 10s
        provider: generic
        ref:
          commit: << inputs.sha >>
        url: https://github.com/developer-friendly/fluxy-dummy
    - apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: fluxy-dummy-pr<< inputs.id >>
        namespace: staging
      spec:
        force: false
        images:
          - name: ghcr.io/developer-friendly/fluxy-dummy
            newTag: << inputs.sha >>
        interval: 5s
        nameSuffix: -pr<< inputs.id >>
        path: kustomize/overlays/preview
        postBuild:
          substitute:
            PR_NUMBER: pr<< inputs.id >>
        prune: true
        sourceRef:
          kind: GitRepository
          name: fluxy-dummy-pr<< inputs.id >>
          namespace: staging
        suspend: false
        targetNamespace: staging
        timeout: 10s
        wait: true
    - apiVersion: gateway.networking.k8s.io/v1
      kind: HTTPRoute
      metadata:
        name: fluxy-dummy-pr<< inputs.id >>
        namespace: staging
      spec:
        hostnames:
          - pr<< inputs.id >>.developer-friendly.blog
        parentRefs:
          - group: gateway.networking.k8s.io
            kind: Gateway
            name: cilium
            namespace: cert-manager
            sectionName: https
        rules:
          - backendRefs:
              - kind: Service
                name: fluxy-dummy-pr<< inputs.id >>
                port: 80
            filters:
              - responseHeaderModifier:
                  set:
                    - name: Strict-Transport-Security
                      value: max-age=31536000; includeSubDomains; preload
                type: ResponseHeaderModifier
            matches:
              - path:
                  type: PathPrefix
                  value: /

You will quickly notice that unlike your typical templating language, the values are populated with << and >>. This is to avoid conflicting variables with Helm templates, in case you create one.

The above three resources specified in the ResourceSet.spec.resources will generate something like the following:

Click to expand 
---
apiVersion: v1
kind: List
items:
  - apiVersion: source.toolkit.fluxcd.io/v1
    kind: GitRepository
    metadata:
      creationTimestamp: "2025-03-09T04:50:38Z"
      finalizers:
        - finalizers.fluxcd.io
      generation: 1
      labels:
        resourceset.fluxcd.controlplane.io/name: fluxy-dummy
        resourceset.fluxcd.controlplane.io/namespace: staging
      name: fluxy-dummy-pr3
      namespace: staging
      resourceVersion: "15283523"
      uid: 0ee1ed82-091a-4e90-b7a3-f56bb3e6b8e5
    spec:
      interval: 10s
      provider: generic
      ref:
        commit: 59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e
      timeout: 60s
      url: https://github.com/developer-friendly/fluxy-dummy
    status:
      artifact:
        digest: sha256:2ef812f74564d4e9617e20d0cb098d172f38423342458b47b15b13406d4cfb43
        lastUpdateTime: "2025-03-09T04:50:48Z"
        path: gitrepository/staging/fluxy-dummy-pr3/59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e.tar.gz
        revision: sha1:59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e
        size: 5805562
        url: http://source-controller.flux-system.svc.cluster.local./gitrepository/staging/fluxy-dummy-pr3/59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e.tar.gz
      conditions:
        - lastTransitionTime: "2025-03-09T04:50:48Z"
          message: stored artifact for revision 'sha1:59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e'
          observedGeneration: 1
          reason: Succeeded
          status: "True"
          type: Ready
        - lastTransitionTime: "2025-03-09T04:50:48Z"
          message: stored artifact for revision 'sha1:59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e'
          observedGeneration: 1
          reason: Succeeded
          status: "True"
          type: ArtifactInStorage
      observedGeneration: 1
  - apiVersion: kustomize.toolkit.fluxcd.io/v1
    kind: Kustomization
    metadata:
      creationTimestamp: "2025-03-09T04:50:38Z"
      finalizers:
        - finalizers.fluxcd.io
      generation: 11
      labels:
        resourceset.fluxcd.controlplane.io/name: fluxy-dummy
        resourceset.fluxcd.controlplane.io/namespace: staging
      name: fluxy-dummy-pr3
      namespace: staging
      resourceVersion: "15342430"
      uid: e10da022-5c17-40c3-b06b-b6a2612e93a4
    spec:
      force: false
      images:
        - name: ghcr.io/developer-friendly/fluxy-dummy
          newTag: 59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e
      interval: 5s
      nameSuffix: -pr3
      path: kustomize/overlays/preview
      postBuild:
        substitute:
          PR_NUMBER: pr3
      prune: true
      sourceRef:
        kind: GitRepository
        name: fluxy-dummy-pr3
        namespace: staging
      suspend: false
      targetNamespace: staging
      timeout: 10s
      wait: true
    status:
      conditions:
        - lastTransitionTime: "2025-03-09T07:23:49Z"
          message: "Applied revision: sha1:59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e"
          observedGeneration: 11
          reason: ReconciliationSucceeded
          status: "True"
          type: Ready
        - lastTransitionTime: "2025-03-09T07:23:49Z"
          message: Health check passed in 130.960449ms
          observedGeneration: 11
          reason: Succeeded
          status: "True"
          type: Healthy
      inventory:
        entries:
          - id: staging_fluxy-dummy-pr3__ServiceAccount
            v: v1
          - id: staging_fluxy-dummy-pr3__Service
            v: v1
          - id: staging_fluxy-dummy-pr3_apps_Deployment
            v: v1
      lastAppliedRevision: sha1:59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e
      lastAttemptedRevision: sha1:59cac9a9b1537f1dc7f4d2a30e26f07379ef7b5e
      lastHandledReconcileAt: "1741502463"
      observedGeneration: 11
  - apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      creationTimestamp: "2025-03-09T04:50:38Z"
      generation: 1
      labels:
        resourceset.fluxcd.controlplane.io/name: fluxy-dummy
        resourceset.fluxcd.controlplane.io/namespace: staging
      name: fluxy-dummy-pr3
      namespace: staging
      resourceVersion: "15338897"
      uid: a602bd3c-6e40-44c6-b658-933fe80f8852
    spec:
      hostnames:
        - pr3.developer-friendly.blog
      parentRefs:
        - group: gateway.networking.k8s.io
          kind: Gateway
          name: cilium
          namespace: cert-manager
          sectionName: https
      rules:
        - backendRefs:
            - group: ""
              kind: Service
              name: fluxy-dummy-pr3
              port: 80
              weight: 1
          filters:
            - responseHeaderModifier:
                set:
                  - name: Strict-Transport-Security
                    value: max-age=31536000; includeSubDomains; preload
              type: ResponseHeaderModifier
          matches:
            - path:
                type: PathPrefix
                value: /
    status:
      parents:
        - conditions:
            - lastTransitionTime: "2025-03-09T07:15:19Z"
              message: Accepted HTTPRoute
              observedGeneration: 1
              reason: Accepted
              status: "True"
              type: Accepted
            - lastTransitionTime: "2025-03-09T04:50:49Z"
              message: Service reference is valid
              observedGeneration: 1
              reason: ResolvedRefs
              status: "True"
              type: ResolvedRefs
          controllerName: io.cilium/gateway-controller
          parentRef:
            group: gateway.networking.k8s.io
            kind: Gateway
            name: cilium
            namespace: cert-manager
            sectionName: https

Making a Change and Create Pull Request

Everything is ready. Let's create a pull request and verify the setup.

In fact, let's create two pull requests adding two separate routes to our web application.

First pull request (pr3):

10-app/main.go
package main

import (
    "net/http"

    "github.com/gin-gonic/gin"
)

func main() {
    r := gin.Default()

    r.GET("/", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "message": "Hello, World!",
        })
    })

    r.GET("/ping", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "message": "pong",
        })
    })

    r.Run()
}

Second pull request (pr4):

10-app/main.go
package main

import (
    "net/http"

    "github.com/gin-gonic/gin"
)

func main() {
    r := gin.Default()

    r.GET("/ping", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "message": "pong",
        })
    })

    r.GET("/hello", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "message": "Hello to you too!",
        })
    })

    r.Run()
}

Now, pushing these changes to the repository, creating a pull request per each branch and labeling them according to our configuration, we should see the preview environments being created.

GitHub Pull Requests
GitHub Pull Requests

We can verify this by checking the corresponding FluxCD resource:

---
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
  annotations:
    fluxcd.controlplane.io/reconcileEvery: 10s
  creationTimestamp: "2025-03-08T07:12:24Z"
  finalizers:
    - fluxcd.controlplane.io/finalizer
  generation: 2
  name: preview-deployments
  namespace: staging
  resourceVersion: "14958793"
  uid: cd84d004-c37d-4eda-883f-ce6ff0900710
spec:
  filter:
    labels:
      - preview
  secretRef:
    name: github-auth
  type: GitHubPullRequest
  url: https://github.com/developer-friendly/fluxy-dummy
status:
  conditions:
    - lastTransitionTime: "2025-03-08T08:34:40Z"
      message: Reconciliation finished in 261ms
      observedGeneration: 2
      reason: ReconciliationSucceeded
      status: "True"
      type: Ready
  exportedInputs:
    - author: meysam81
      branch: meysam/feat/another-route
      id: "4"
      sha: 5a39668b2b7394a76a7ccbdf3364919ef12506a0
      title: "feat: add another route"
    - author: meysam81
      branch: meysam/feat/index-route
      id: "3"
      sha: b449283e4736a271788b7a4e7689fd7a49c279ff
      title: "feat: add index route"
  lastExportedRevision: sha256:d013751ec78662d8c89c0bb398f9715c4ae8309631e43808c37663eacd8fefb8
  lastHandledReconcileAt: "1741421676"

Notice the exportedInputs per each of the detected pull requests. We have 5 inputs so far, providing the values we have specified in our ResourceSet CRD.

Let's verify the deployed instances:

$ curl https://pr3.developer-friendly.blog/ -D -


HTTP/2 200
date: Sat, 08 Mar 2025 09:02:37 GMT
content-type: application/json; charset=utf-8
content-length: 27
x-envoy-upstream-service-time: 0
strict-transport-security: max-age=15552000; includeSubDomains; preload
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=ogzBE6LUJ2LTjON%2FW7%2Fhes8VBPZABnT9gqLx07b30%2FmZH6Z7BgIm7mFae509jeya%2BOktiefT6fsoDx6yiahQOdlBcY9qFmb6%2Bt3VfK548i02s%2BokGLc3%2FseKGep1Nj35zgoUMr9xHyJJsOc%2F0%2FI%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
expect-ct: max-age=86400, enforce
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: cloudflare
cf-ray: 91d12798385f4900-SIN
alt-svc: h3=":443"; ma=86400
server-timing: cfL4;desc="?proto=TCP&rtt=87909&min_rtt=82834&rtt_var=30301&sent=6&recv=8&lost=0&retrans=0&sent_bytes=3439&recv_bytes=874&delivery_rate=34961&cwnd=172&unsent_bytes=0&cid=e015ecd88c5a5cc5&ts=642&x=0"

{"message":"Hello, World!"}

$ curl https://pr4.developer-friendly.blog/hello -D -


HTTP/2 200
date: Sat, 08 Mar 2025 09:02:31 GMT
content-type: application/json; charset=utf-8
content-length: 31
x-envoy-upstream-service-time: 0
strict-transport-security: max-age=15552000; includeSubDomains; preload
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=70t384ON8YOvLnAKU6%2Bo9%2BoLAhqjcnpjIz2lN6Va4eY1nDPShUtMKfLJPnwpzuRjGo6R%2FwNxy5p6woXODm6brXx5SK5xoln%2B67oKce63xNcxM8AeU4qab%2FoRpgTM1b9pG0%2BrFUEBABQjXuacw2Y%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
speculation-rules: "/cdn-cgi/speculation"
expect-ct: max-age=86400, enforce
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: cloudflare
cf-ray: 91d127765f9440fb-SIN
alt-svc: h3=":443"; ma=86400
server-timing: cfL4;desc="?proto=TCP&rtt=94988&min_rtt=85107&rtt_var=28063&sent=7&recv=8&lost=0&retrans=0&sent_bytes=3460&recv_bytes=880&delivery_rate=31641&cwnd=253&unsent_bytes=0&cid=a8621ada7c67c31f&ts=638&x=0"

{"message":"Hello to you too!"}

It works. Great.

Comment the Preview Environment URL

One last objective before we close this off. Let's comment on the corresponding pull request the URL of the preview environment.

We want each pull request to do this automatically. Think of it as a one-off task that executes to completion.

There are more than one way to achive this, but the objective for us is to use an initContainer in our application that will run right before our main container starts.

20-infra/40-preview-environment/rset.yml
---
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: fluxy-dummy
  namespace: staging
  annotations:
    fluxcd.controlplane.io/reconcile: "enabled"
    fluxcd.controlplane.io/reconcileEvery: "10s"
    fluxcd.controlplane.io/reconcileTimeout: "1m"
spec:
  serviceAccountName: flux
  inputsFrom:
    - apiVersion: fluxcd.controlplane.io/v1
      kind: ResourceSetInputProvider
      name: preview-deployments
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      metadata:
        name: fluxy-dummy-pr<< inputs.id >>
        namespace: staging
      spec:
        interval: 10s
        provider: generic
        ref:
          commit: << inputs.sha >>
        url: https://github.com/developer-friendly/fluxy-dummy
    - apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: fluxy-dummy-pr<< inputs.id >>
        namespace: staging
      spec:
        force: false
        images:
          - name: ghcr.io/developer-friendly/fluxy-dummy
            newTag: << inputs.sha >>
        interval: 5s
        nameSuffix: -pr<< inputs.id >>
        patches:
          - patch: |
              - op: add
                path: /spec/template/spec/initContainers/-
                value:
                  args:
                    - developer-friendly/fluxy-dummy
                  env:
                    - name: USER_LOGIN
                      valueFrom:
                        secretKeyRef:
                          key: username
                          name: github-auth
                          optional: false
                    - name: GITHUB_TOKEN
                      valueFrom:
                        secretKeyRef:
                          key: password
                          name: github-auth
                          optional: false
                    - name: COMMIT_SHA
                      value: << inputs.sha >>
                    - name: PR_NUMBER
                      value: "<< inputs.id >>"
                    - name: URL
                      value: https://pr<< inputs.id >>.developer-friendly.blog
                  image: ghcr.io/meysam81/preview-bot:v1.0.13
                  name: preview-bot
                  resources:
                    limits:
                      cpu: 10m
                      memory: 10Mi
                    requests:
                      cpu: 10m
                      memory: 10Mi
                  securityContext:
                    allowPrivilegeEscalation: false
                    capabilities:
                      drop:
                        - ALL
                    readOnlyRootFilesystem: true
                    runAsGroup: 65534
                    runAsNonRoot: true
                    runAsUser: 65534
                  terminationMessagePolicy: FallbackToLogsOnError
            target:
              kind: Deployment
              name: fluxy-dummy
        path: kustomize/overlays/preview
        postBuild:
          substitute:
            PR_NUMBER: pr<< inputs.id >>
        prune: true
        sourceRef:
          kind: GitRepository
          name: fluxy-dummy-pr<< inputs.id >>
          namespace: staging
        suspend: false
        targetNamespace: staging
        timeout: 10s
        wait: true
    - apiVersion: gateway.networking.k8s.io/v1
      kind: HTTPRoute
      metadata:
        name: fluxy-dummy-pr<< inputs.id >>
        namespace: staging
      spec:
        hostnames:
          - pr<< inputs.id >>.developer-friendly.blog
        parentRefs:
          - group: gateway.networking.k8s.io
            kind: Gateway
            name: cilium
            namespace: cert-manager
            sectionName: https
        rules:
          - backendRefs:
              - kind: Service
                name: fluxy-dummy-pr<< inputs.id >>
                port: 80
            filters:
              - responseHeaderModifier:
                  set:
                    - name: Strict-Transport-Security
                      value: max-age=31536000; includeSubDomains; preload
                type: ResponseHeaderModifier
            matches:
              - path:
                  type: PathPrefix
                  value: /

An improvement to this decision would possibly be to run it as a lifecycle.postStart in the pod spec10, but that means adding another binary to our main application!

Here's what the message looks like11:

Preivew Bot Comment
Preivew Bot Comment

Conclusion

That's all folks!

The main mission in this blog post was to create preview environment on each pull request of our GitHub repository and inform the author via a comment.

The current setup works flawlessly as scoped out at the beginning of this blog post and can give you inspiration to setup your own preview environments.

This is not my first time provisioning preview environments in Kubernetes. I have achieved the same with ArgoCD, GitHub Actions, etc.

But to tell you the truth, FluxCD holds a close place in my heart. It's simple, decoupled, and will always be my first GitOps tool of choice.

Until next time 🫡, ciao 🤠 & happy coding! 🐧

Subscribe to Newsletter Subscribe to RSS Feed

Share on Share on Share on Share on

  1. https://www.gitpod.io/blog/we-are-leaving-kubernetes ↩

  2. https://fluxcd.io/flux/components/notification/receivers/ ↩

  3. https://fluxcd.io/flux/installation/ ↩

  4. https://github.com/helm/helm/releases/tag/v3.17.1 ↩

  5. https://fluxcd.io/flux/components/kustomize/kustomizations/#post-build-variable-substitution ↩

  6. https://kubernetes.io/docs/concepts/services-networking/service/ ↩

  7. https://fluxcd.control-plane.io/operator/resourcesets/github-pull-requests/#preview-namespace ↩

  8. https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys ↩

  9. https://fluxcd.control-plane.io/operator/resourcesets/github-pull-requests/#github-webhook ↩

  10. https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#lifecycle-1 ↩

  11. https://github.com/developer-friendly/fluxy-dummy/pull/3#issuecomment-2708701125 ↩

Comments