Skip to content

How to Install Lightweight Kubernetes on Ubuntu 22.04

Learn how to deploy a lightweight Kubernetes cluster using k3s on Ubuntu 22.04 using OpenTofu & Ansible on Hetzner Cloud with Cilium as the CNI.

Introduction

Kubernetes is a powerful container orchestration platform that allows you to deploy, scale, and manage containerized applications. However, setting up a Kubernetes cluster can be a complex and time-consuming process and usually requires a PhD in Kubernetes.

However, most people are not in the business of managing Kubernetes clusters, nor do they feel nerdy enough to spend their weekends configuring YAML files.

Complexity not an issue for you?

If you're like me, enjoying a good challenge and have a knack for tackling complexity, you will be greatly served by spinning up Kubernetes the Hard Way I have published recently.

On the other hand, it's not just about the level of complexity involved in managing a bunch of Kubernetes components, but also the cost of running a full-fledged Kubernetes cluster in the cloud, be it managed or self-hosted.

That is to say that you will likely find yourself looking for a robust and production-ready Kubernetes distribution that is lightweight, easy to install, and easy to manage.

Disclaimer

This post is influenced by a community article1 on the Hetzner Tutorials. I wanted to add Cilium and Ansible to the mix, so I started my own journey. I hope you find this post helpful.

Why should you care?

That's where k3s comes into play. k3s is a lightweight Kubernetes distribution that is designed for production workloads, resource-constrained environments, and edge computing. It is a fully compliant Kubernetes distribution that is packaged in a single binary and requires minimal dependencies.

In this post, I will show you how to install k3s on Ubuntu 22.04 using Hetzner Cloud, OpenTofu, Ansible, and Cilium. Stay with me till the end cause we got some cool stuff to cover.

Prerequisites

Here are the list of CLI tools you need installed on your machine.

Step 0: Generate Hetzner API token

For the purpose of this tutorial, we'll only spin up a single server. However, for production workloads, you should consider setting up a multi-node cluster for high availability and fault tolerance. You will be guided with some of the obvious tips at the end of this post.

First, you need to create an account on Hetzner Cloud2 and generate an API token3. Your generated token must have write access cause the TF files will create resources on your behalf.

Step 1: Create the server

Let us structure our project directory as follows:

Text Only
.
├── ansible/
└── opentofu/

Now, let's pin the version for the required TF provider.

opentofu/versions.tf
terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1.45"
    }
  }

  required_version = "<2"
}

variable "hetzner_api_token" {
  type      = string
  nullable  = false
  sensitive = true
}

provider "hcloud" {
  token = var.hetzner_api_token
}
Provider Versioning

The pinned version in this example is very loose. You might want to be more specific in scalable environments where you have multiple projects and need to ensure nothing breaks before you intentionally upgrade with enough tests.

As an example, you can see the Rust infrastructure team4 being too conservative in their Ansible versioning as they don't want team members get caught off guard with a breaking change.

Notice that we have also created a variable to pass in the previously generated Hetzner API token. This is a good practice to avoid hardcoding sensitive information in your codebase.

To efficiently benefit from this technique, you can add the following lines to your .bashrc file for all the future projects.

~/.bashrc
# ... truncated ...

export TF_VAR_hetzner_api_token="PLACEHOLDER"

Next, we'll define the creation of the Hetzner Cloud Server in our TF file.

opentofu/main.tf
resource "hcloud_server" "this" {
  name        = "k3s-cluster"
  server_type = "cax11" # ARM64, 2vCPU, 4GB RAM
  image       = "ubuntu-22.04"
  location    = "nbg1" # Nuremberg DC

  user_data = file("${path.module}/cloud-init.yml")
}

I don't know about you, but I personally love ARM processors. They are energy efficient and have a great performance. Whenever possible, I will always opt-in for ARM processors.

ARM64 Architecture

You got to consider your workload before adopting ARM64 architecture. Not all softwares are compatible with ARM64. However, the good news is that most of the popular softwares are already compatible with ARM64, including the k3s we are going to install in this tutorial.

You might have noticed in line 7 that we are passing in a cloud-init5 file. That's an important piece of the whole puzzle. Let's create that file.

Generating SSH key pair

If you don't have an SSH key pair, you can generate one using the following command:

Password-less elliptic curve key pair
ssh-keygen -t ed25519 -f ~/.ssh/k3s-cluster.hetzner -N ''
opentofu/cloud-init.yml
#cloud-config
users:
  - name: k8s
    groups: users, admin
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - # Place the hardcoded value of your SSH public key here
        # Output of `ssh-keygen` .pub file
packages:
  - fail2ban
  - python3
  - python3-pip
package_update: true
package_upgrade: true
runcmd:
  # Allow reading the logs using journalctl
  - usermod -aG adm k8s
  # Ban malicious IPs from overly aggressive SSH login attempts
  - printf "[sshd]\nenabled = true\nbanaction = iptables-multiport" > /etc/fail2ban/jail.local
  - systemctl enable fail2ban
  # Configure SSH daemon in a more secure way
  - sed -i -e '/^\(#\|\)PermitRootLogin/s/^.*$/PermitRootLogin no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)PasswordAuthentication/s/^.*$/PasswordAuthentication no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)KbdInteractiveAuthentication/s/^.*$/KbdInteractiveAuthentication no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)ChallengeResponseAuthentication/s/^.*$/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)MaxAuthTries/s/^.*$/MaxAuthTries 2/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)AllowTcpForwarding/s/^.*$/AllowTcpForwarding no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)X11Forwarding/s/^.*$/X11Forwarding no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)AllowAgentForwarding/s/^.*$/AllowAgentForwarding no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)AuthorizedKeysFile/s/^.*$/AuthorizedKeysFile .ssh\/authorized_keys/' /etc/ssh/sshd_config
  - sed -i '$a AllowUsers k8s' /etc/ssh/sshd_config
  # Install the Kubernetes cluster using k3s
  - |
    curl https://get.k3s.io | \
      INSTALL_K3S_VERSION="v1.29.2+k3s1" \
      INSTALL_K3S_EXEC="--disable traefik
        --disable-network-policy
        --flannel-backend none
        --write-kubeconfig /home/k8s/.kube/config
        --secrets-encryption" \
      sh -
  - chown -R k8s:k8s /home/k8s/.kube/
  # Ensure all the settings are applied
  - reboot

The first line of this file is not a normal comment. It's a directive that tells the cloud-init to ignore the rest of the file if it's not a valid cloud-init config. It has a similar behavior to the shebang in shell scripts.

Shebang

The shebang is a special comment that tells the shell which interpreter to use to execute the script. It's usually the first line of a script file.

hello.sh
#!/bin/bash
echo "Hello, World!"

As soon as I make this script executable using chmod +x hello.sh, I can run it using ./hello.sh and by parsing the first line, the shell knows that it should use the bash interpreter to execute the script.

Explain the k3s installation

All the config in cloud-init file is self-explanatory. However, I'd like to highlight a few important points.

Config/Flag Description
--disable-network-policy This flag disables the default network policy controller in k3s. We will use Cilium as the CNI plugin, which comes with its own network policy controller.
--flannel-backend none This flag disables the default Flannel CNI plugin in k3s. Again, we will use Cilium as the CNI plugin.
--secrets-encryption This flag enables the secrets encryption feature in k3s. This is a security feature that encrypts the secrets stored in etcd.
--disable traefik We will not require Traefik as an Ingress Controller in our setup. We will use Cilium with the help of ServiceLB of k3s.

Finally, let's create the relevant output that we'll use for later steps.

opentofu/output.tf
output "server_ipv4_address" {
  value = hcloud_server.this.ipv4_address
}

output "server_ipv6_address" {
  value = hcloud_server.this.ipv6_address
}

We have our TF files ready for provisioning the server. Let's create the server using the tofu CLI.

cd opentofu
tofu init
tofu plan -out tfplan
# observe the plan visually and confirm the changes
tofu apply tfplan

Now that we've got the IP address, we can use either the IPv4 or the IPv6 to connect to the server.

IP_ADDRESS=$(tofu output -raw server_ipv4_address)
echo "${IP_ADDRESS} k3s-cluster.hetzner" | sudo tee -a /etc/hosts

This will ensure that the hostname k3s-cluster.hetzner resolves to the IP address of the server and we won't have to memorize the IP address for every command we run.

Step 1.1: Prepare Ansible Inventory

We will heavily rely on the SSH connections to the machine as the default way of connection for Ansible. Therefore, to make that access easier and password-less, we gotta take two steps:

  1. Prepare the SSH config file
  2. Prepare the Ansible inventory file

Let's start with the SSH config file.

~/.ssh/config
# ... truncated ...

Host k3s-cluster.hetzner
  User k8s
  IdentityFile ~/.ssh/k3s-cluster.hetzner # location of `ssh-keygen` command

Now, let's prepare the Ansible inventory file using the yaml plugin of Ansible inventory.

ansible/inventory.yml
k8s:
  hosts:
    k3s-cluster:
      ansible_host: k3s-cluster.hetzner

Step 2: Bootstrap the cluster

So far, we have provisioned the cluster using OpenTofu. But, if you SSH into the machine, kubectl get nodes will return a NotReady state for your node. That is because of the absence of a CNI plugin in the cluster.

It's time to use Ansible to take care of that, although, arguably, we could have done that in the cloud-init file as well.

But, using Ansible gives more flexibility and control over the steps involved and it will make it reproducible upon future invocations.

ansible/playbook.yml
- name: Bootstrap the Kubernetes cluster
  hosts: all
  gather_facts: true
  become: true
  vars:
    helm_version: v3.14.3
  tasks:
    - name: Install Kubernetes library
      ansible.builtin.pip:
        name: kubernetes<30
        state: present
    - name: Install helm binary
      ansible.builtin.shell:
        cmd: "{{ lookup('ansible.builtin.url', 'https://git.io/get_helm.sh', split_lines=false) }}"
        creates: /usr/local/bin/helm
      environment:
        DESIRED_VERSION: "{{ helm_version }}"
    - name: Install Kubernetes gateway CRDs
      kubernetes.core.k8s:
        src: "{{ item }}"
        state: present
      loop:
        - https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
        - https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
    - name: Install cilium
      block:
        - name: Add cilium helm repository
          kubernetes.core.helm_repository:
            name: cilium
            repo_url: https://helm.cilium.io
        - name: Install cilium helm release
          kubernetes.core.helm:
            name: cilium
            chart_ref: cilium/cilium
            namespace: kube-system
            state: present
            chart_version: 1.15.2
            values:
              gatewayAPI:
                enabled: true
              kubeProxyReplacement: true
              encryption:
                enabled: true
                type: wireguard
              operator:
                replicas: 1

As you see in the highlighted lines, we will use Kubernetes Gateway API6 as a replacement for Ingress Controller. This is a more modern approach and it has more capabilities than the traditional Ingress Controller.

Among many other benefits, Gateway API has the ability to handle TLS, gRPC, and WebSockets, which are not supported by Ingress Controller.

Why Gateway API?

These days I will use Gateway API for all my Kubernetes clusters, unless I have a specific requirement that mandates the use of Ingress Controller.

Cilium has native support7 for Gatway API.

To run this ansible playbook, I will simply run the following command:

ansible-playbook -i ansible/inventory.yml ansible/playbook.yml -v

You might as well ignore that last -v flag, but I like to see the output of the playbook as it runs.

Ansible Configuration

In fact, I have a global user configuration that has some sane defaults. Completely contradictory to what the Ansible sane defaults are.

~/.ansible.cfg
[defaults]
become=false
log_path=/tmp/ansible.log
gather_facts=false
fact_caching = ansible.builtin.jsonfile
fact_caching_connection = /tmp/ansible_facts
cache_timeout = 3600
interpreter_python = auto_silent
verbosity = 2
ssh_common_args = -o ConnectTimeout=5

[inventory]
enable_plugins = 'host_list', 'script', 'auto', 'yaml', 'ini', 'toml', 'azure_rm', 'aws_ec2', 'auto'
cache = yes
cache_connection = /tmp/ansible_inventory

Step 3: Use the Cluster

We have done all the heavy lifting thus far and now it's time to use the cluster.

For that, you can either use scp or rsync to fetch the ~/.kube/config into your local machine, or use the kubectl command right from the remote server.

I generally prefer the former as I believe the control machine is my localhost and everything else is a remote machine that hosts the workload I command it to.

NOTE: In the case where you copy the remote Kubeconfig file to your local machine, you will need to update the server field in the ~/.kube/config file as it is pointing to the 127.0.0.1 by default.

~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:6443 # CHANGE THIS LINE
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED

Step 4: Protect the Server

Since exposing the Kubernetes API server means the cluster shall be internet-accessible, I, as a security-conscious person, would protect my server with the help of firewall.

That said, I would, as a last step, deploy the free Hetzner Cloud Firewall to protect my server.

opentofu/versions.tf
terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1.45"
    }
    http = {
      source  = "hashicorp/http"
      version = "~> 3.4"
    }
  }

  required_version = "<2"
}

variable "hetzner_api_token" {
  type      = string
  nullable  = false
  sensitive = true
}

provider "hcloud" {
  token = var.hetzner_api_token
}
opentofu/network.tf
data "http" "this" {
  url = "https://api.ipify.org"
}

resource "hcloud_firewall" "this" {
  name = "k3s-cluster"

  dynamic "rule" {
    for_each = toset({
      ssh            = 22
      kubernetes_api = 6443
    })
    content {
      direction = "in"
      protocol  = "tcp"
      port      = rule.value
      source_ips = [
        format("%s/32", data.http.this.response_body),
      ]
      description = "Admin public IP address"
    }
  }
}

resource "hcloud_firewall_attachment" "this" {
  firewall_id = hcloud_firewall.this.id
  server_ids  = [hcloud_server.this.id]
}

Bonus: Multi-Node Cluster

When planning to go for production in this setup, you are advised to go for a multi-node deployment for high availability and fault tolerance.

To achieve that, you can pass the K3S_URL8 and K3S_TOKEN to the cloud-init script for any of the worker nodes.

Source Code

All the code examples in this post are publicly available9 on GitHub under the Apache 2.0 license.

If you enjoyed this blog post, consider sharing it with these buttons 👇. Please leave a comment for us at the end, we read & love 'em all.. ❣

Share on Share on Share on Share on

Comments