How to Install Lightweight Kubernetes on Ubuntu 22.04¶

Learn how to deploy a lightweight Kubernetes cluster using k3s on Ubuntu 22.04 using OpenTofu & Ansible on Hetzner Cloud with Cilium as the CNI.

Introduction¶

Kubernetes is a powerful container orchestration platform that allows you to deploy, scale, and manage containerized applications. However, setting up a Kubernetes cluster can be a complex and time-consuming process and usually requires a PhD in Kubernetes.

However, most people are not in the business of managing Kubernetes clusters, nor do they feel nerdy enough to spend their weekends configuring YAML files.

On the other hand, it's not just about the level of complexity involved in managing a bunch of Kubernetes components, but also the cost of running a full-fledged Kubernetes cluster in the cloud, be it managed or self-hosted.

That is to say that you will likely find yourself looking for a robust and production-ready Kubernetes distribution that is lightweight, easy to install, and easy to manage.

Why should you care?¶

That's where k3s comes into play. k3s is a lightweight Kubernetes distribution that is designed for production workloads, resource-constrained environments, and edge computing. It is a fully compliant Kubernetes distribution that is packaged in a single binary and requires minimal dependencies.

In this post, I will show you how to install k3s on Ubuntu 22.04 using Hetzner Cloud, OpenTofu, Ansible, and Cilium. Stay with me till the end cause we got some cool stuff to cover.

Prerequisites¶

Here are the list of CLI tools you need installed on your machine.

• OpenTofu v1.6
• Ansible v2.16

Step 0: Generate Hetzner API token¶

For the purpose of this tutorial, we'll only spin up a single server. However, for production workloads, you should consider setting up a multi-node cluster for high availability and fault tolerance. You will be guided with some of the obvious tips at the end of this post.

First, you need to create an account on Hetzner Cloud² and generate an API token³. Your generated token must have write access cause the TF files will create resources on your behalf.

Step 1: Create the server¶

Let us structure our project directory as follows:

.
├── ansible/
└── opentofu/

Now, let's pin the version for the required TF provider.

terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1.45"
    }
  }

  required_version = "<2"
}

variable "hetzner_api_token" {
  type      = string
  nullable  = false
  sensitive = true
}

provider "hcloud" {
  token = var.hetzner_api_token
}

Notice that we have also created a variable to pass in the previously generated Hetzner API token. This is a good practice to avoid hardcoding sensitive information in your codebase.

To efficiently benefit from this technique, you can add the following lines to your .bashrc file for all the future projects.

# ... truncated ...

export TF_VAR_hetzner_api_token="PLACEHOLDER"

Next, we'll define the creation of the Hetzner Cloud Server in our TF file.

resource "hcloud_server" "this" {
  name        = "k3s-cluster"
  server_type = "cax11" # ARM64, 2vCPU, 4GB RAM
  image       = "ubuntu-22.04"
  location    = "nbg1" # Nuremberg DC

  user_data = file("${path.module}/cloud-init.yml")
}

I don't know about you, but I personally love ARM processors. They are energy efficient and have a great performance. Whenever possible, I will always opt-in for ARM processors.

You might have noticed in line 7 that we are passing in a cloud-init⁵ file. That's an important piece of the whole puzzle. Let's create that file.

ssh-keygen -t ed25519 -f ~/.ssh/k3s-cluster.hetzner -N ''

#cloud-config
users:
  - name: k8s
    groups: users, admin
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - # Place the hardcoded value of your SSH public key here
        # Output of `ssh-keygen` .pub file
packages:
  - fail2ban
  - python3
  - python3-pip
package_update: true
package_upgrade: true
runcmd:
  # Allow reading the logs using journalctl
  - usermod -aG adm k8s
  # Ban malicious IPs from overly aggressive SSH login attempts
  - printf "[sshd]\nenabled = true\nbanaction = iptables-multiport" > /etc/fail2ban/jail.local
  - systemctl enable fail2ban
  # Configure SSH daemon in a more secure way
  - sed -i -e '/^\(#\|\)PermitRootLogin/s/^.*$/PermitRootLogin no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)PasswordAuthentication/s/^.*$/PasswordAuthentication no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)KbdInteractiveAuthentication/s/^.*$/KbdInteractiveAuthentication no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)ChallengeResponseAuthentication/s/^.*$/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)MaxAuthTries/s/^.*$/MaxAuthTries 2/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)AllowTcpForwarding/s/^.*$/AllowTcpForwarding no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)X11Forwarding/s/^.*$/X11Forwarding no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)AllowAgentForwarding/s/^.*$/AllowAgentForwarding no/' /etc/ssh/sshd_config
  - sed -i -e '/^\(#\|\)AuthorizedKeysFile/s/^.*$/AuthorizedKeysFile .ssh\/authorized_keys/' /etc/ssh/sshd_config
  - sed -i '$a AllowUsers k8s' /etc/ssh/sshd_config
  # Install the Kubernetes cluster using k3s
  - |
    curl https://get.k3s.io | \
      INSTALL_K3S_VERSION="v1.29.2+k3s1" \
      INSTALL_K3S_EXEC="--disable traefik
        --disable-network-policy
        --flannel-backend none
        --write-kubeconfig /home/k8s/.kube/config
        --secrets-encryption" \
      sh -
  - chown -R k8s:k8s /home/k8s/.kube/
  # Ensure all the settings are applied
  - reboot

The first line of this file is not a normal comment. It's a directive that tells the cloud-init to ignore the rest of the file if it's not a valid cloud-init config. It has a similar behavior to the shebang in shell scripts.

#!/bin/bash
echo "Hello, World!"

Explain the k3s installation¶

All the config in cloud-init file is self-explanatory. However, I'd like to highlight a few important points.

Finally, let's create the relevant output that we'll use for later steps.

output "server_ipv4_address" {
  value = hcloud_server.this.ipv4_address
}

output "server_ipv6_address" {
  value = hcloud_server.this.ipv6_address
}

We have our TF files ready for provisioning the server. Let's create the server using the tofu CLI.

cd opentofu
tofu init
tofu plan -out tfplan
# observe the plan visually and confirm the changes
tofu apply tfplan

Now that we've got the IP address, we can use either the IPv4 or the IPv6 to connect to the server.

IP_ADDRESS=$(tofu output -raw server_ipv4_address)
echo "${IP_ADDRESS} k3s-cluster.hetzner" | sudo tee -a /etc/hosts

This will ensure that the hostname k3s-cluster.hetzner resolves to the IP address of the server and we won't have to memorize the IP address for every command we run.

Step 1.1: Prepare Ansible Inventory¶

We will heavily rely on the SSH connections to the machine as the default way of connection for Ansible. Therefore, to make that access easier and password-less, we gotta take two steps:

1. Prepare the SSH config file
2. Prepare the Ansible inventory file

Let's start with the SSH config file.

# ... truncated ...

Host k3s-cluster.hetzner
  User k8s
  IdentityFile ~/.ssh/k3s-cluster.hetzner # location of `ssh-keygen` command

Now, let's prepare the Ansible inventory file using the yaml plugin of Ansible inventory.

k8s:
  hosts:
    k3s-cluster:
      ansible_host: k3s-cluster.hetzner

Step 2: Bootstrap the cluster¶

So far, we have provisioned the cluster using OpenTofu. But, if you SSH into the machine, kubectl get nodes will return a NotReady state for your node. That is because of the absence of a CNI plugin in the cluster.

It's time to use Ansible to take care of that, although, arguably, we could have done that in the cloud-init file as well.

But, using Ansible gives more flexibility and control over the steps involved and it will make it reproducible upon future invocations.

- name: Bootstrap the Kubernetes cluster
  hosts: all
  gather_facts: true
  become: true
  vars:
    helm_version: v3.14.3
  tasks:
    - name: Install Kubernetes library
      ansible.builtin.pip:
        name: kubernetes<30
        state: present
    - name: Install helm binary
      ansible.builtin.shell:
        cmd: "{{ lookup('ansible.builtin.url', 'https://git.io/get_helm.sh', split_lines=false) }}"
        creates: /usr/local/bin/helm
      environment:
        DESIRED_VERSION: "{{ helm_version }}"
    - name: Install Kubernetes gateway CRDs
      kubernetes.core.k8s:
        src: "{{ item }}"
        state: present
      loop:
        - https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
        - https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
    - name: Install cilium
      block:
        - name: Add cilium helm repository
          kubernetes.core.helm_repository:
            name: cilium
            repo_url: https://helm.cilium.io
        - name: Install cilium helm release
          kubernetes.core.helm:
            name: cilium
            chart_ref: cilium/cilium
            namespace: kube-system
            state: present
            chart_version: 1.15.2
            values:
              gatewayAPI:
                enabled: true
              kubeProxyReplacement: true
              encryption:
                enabled: true
                type: wireguard
              operator:
                replicas: 1

As you see in the highlighted lines, we will use Kubernetes Gateway API⁶ as a replacement for Ingress Controller. This is a more modern approach and it has more capabilities than the traditional Ingress Controller.

Among many other benefits, Gateway API has the ability to handle TLS, gRPC, and WebSockets, which are not supported by Ingress Controller.

Cilium has native support⁷ for Gatway API.

To run this ansible playbook, I will simply run the following command:

ansible-playbook -i ansible/inventory.yml ansible/playbook.yml -v

You might as well ignore that last -v flag, but I like to see the output of the playbook as it runs.

[defaults]
become=false
log_path=/tmp/ansible.log
gather_facts=false
fact_caching = ansible.builtin.jsonfile
fact_caching_connection = /tmp/ansible_facts
cache_timeout = 3600
interpreter_python = auto_silent
verbosity = 2
ssh_common_args = -o ConnectTimeout=5

[inventory]
enable_plugins = 'host_list', 'script', 'auto', 'yaml', 'ini', 'toml', 'azure_rm', 'aws_ec2', 'auto'
cache = yes
cache_connection = /tmp/ansible_inventory

Step 3: Use the Cluster¶

We have done all the heavy lifting thus far and now it's time to use the cluster.

For that, you can either use scp or rsync to fetch the ~/.kube/config into your local machine, or use the kubectl command right from the remote server.

I generally prefer the former as I believe the control machine is my localhost and everything else is a remote machine that hosts the workload I command it to.

NOTE: In the case where you copy the remote Kubeconfig file to your local machine, you will need to update the server field in the ~/.kube/config file as it is pointing to the 127.0.0.1 by default.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:6443 # CHANGE THIS LINE
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED

Step 4: Protect the Server¶

Since exposing the Kubernetes API server means the cluster shall be internet-accessible, I, as a security-conscious person, would protect my server with the help of firewall.

That said, I would, as a last step, deploy the free Hetzner Cloud Firewall to protect my server.

terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1.45"
    }
    http = {
      source  = "hashicorp/http"
      version = "~> 3.4"
    }
  }

  required_version = "<2"
}

variable "hetzner_api_token" {
  type      = string
  nullable  = false
  sensitive = true
}

provider "hcloud" {
  token = var.hetzner_api_token
}

data "http" "this" {
  url = "https://api.ipify.org"
}

resource "hcloud_firewall" "this" {
  name = "k3s-cluster"

  dynamic "rule" {
    for_each = toset({
      ssh            = 22
      kubernetes_api = 6443
    })
    content {
      direction = "in"
      protocol  = "tcp"
      port      = rule.value
      source_ips = [
        format("%s/32", data.http.this.response_body),
      ]
      description = "Admin public IP address"
    }
  }
}

resource "hcloud_firewall_attachment" "this" {
  firewall_id = hcloud_firewall.this.id
  server_ids  = [hcloud_server.this.id]
}

Bonus: Multi-Node Cluster¶

When planning to go for production in this setup, you are advised to go for a multi-node deployment for high availability and fault tolerance.

To achieve that, you can pass the K3S_URL⁸ and K3S_TOKEN to the cloud-init script for any of the worker nodes.

Source Code¶

All the code examples in this post are publicly available⁹ on GitHub under the Apache 2.0 license.