How to Install Lightweight Kubernetes on Ubuntu 22.04¶
Learn how to deploy a lightweight Kubernetes cluster using k3s on Ubuntu 22.04 using OpenTofu & Ansible on Hetzner Cloud with Cilium as the CNI.
How to Install Kubernetes on Ubuntu 22.04¶
Kubernetes is a powerful container orchestration platform that allows you to deploy, scale, and manage containerized applications. However, setting up a Kubernetes cluster can be a complex and time-consuming process and usually requires a PhD in Kubernetes.
However, most people are not in the business of managing Kubernetes clusters, nor do they feel nerdy enough to spend their weekends configuring YAML files.
Complexity not an issue for you?
If you're like me, enjoying a good challenge and have a knack for tackling complexity, you will be greatly served by spinning up Kubernetes the Hard Way I have published recently.
On the other hand, it's not just about the level of complexity involved in managing a bunch of Kubernetes components, but also the cost of running a full-fledged Kubernetes cluster in the cloud, be it managed or self-hosted.
That is to say that you will likely find yourself looking for a robust and production-ready Kubernetes distribution that is lightweight, easy to install, and easy to manage.
Disclaimer
This post is influenced by a community article1 on the Hetzner Tutorials. I wanted to add Cilium and Ansible to the mix, so I started my own journey. I hope you find this post helpful.
Lightweight Kubernetes¶
That's where k3s comes into play. k3s is a lightweight Kubernetes distribution that is designed for production workloads, resource-constrained environments, and edge computing. It is a fully compliant Kubernetes distribution that is packaged in a single binary and requires minimal dependencies.
In this post, I will show you how to install k3s on Ubuntu 22.04 using Hetzner Cloud, OpenTofu, Ansible, and Cilium. Stay with me till the end cause we got some cool stuff to cover.
Prerequisites¶
Here are the list of CLI tools you need installed on your machine.
Step 0: Generate Hetzner API token¶
For the purpose of this tutorial, we'll only spin up a single server. However, for production workloads, you should consider setting up a multi-node cluster for high availability and fault tolerance. You will be guided with some of the obvious tips at the end of this post.
First, you need to create an account on Hetzner Cloud2 and generate an API token3. Your generated token must have write access cause the TF files will create resources on your behalf.
Step 1: Create the server¶
Let us structure our project directory as follows:
Now, let's pin the version for the required TF provider.
terraform {
required_providers {
hcloud = {
source = "hetznercloud/hcloud"
version = "~> 1.45"
}
}
required_version = "<2"
}
variable "hetzner_api_token" {
type = string
nullable = false
sensitive = true
}
provider "hcloud" {
token = var.hetzner_api_token
}
Provider Versioning
The pinned version in this example is very loose. You might want to be more specific in scalable environments where you have multiple projects and need to ensure nothing breaks before you intentionally upgrade with enough tests.
As an example, you can see the Rust infrastructure team4 being too conservative in their Ansible versioning as they don't want team members get caught off guard with a breaking change.
Notice that we have also created a variable to pass in the previously generated Hetzner API token. This is a good practice to avoid hardcoding sensitive information in your codebase.
To efficiently benefit from this technique, you can add the following lines to your .bashrc
file for all the future projects.
Next, we'll define the creation of the Hetzner Cloud Server in our TF file.
resource "hcloud_server" "this" {
name = "k3s-cluster"
server_type = "cax11" # ARM64, 2vCPU, 4GB RAM
image = "ubuntu-22.04"
location = "nbg1" # Nuremberg DC
user_data = file("${path.module}/cloud-init.yml")
}
I don't know about you, but I personally love ARM processors. They are energy efficient and have a great performance. Whenever possible, I will always opt-in for ARM processors.
ARM64 Architecture
You got to consider your workload before adopting ARM64 architecture. Not all softwares are compatible with ARM64. However, the good news is that most of the popular softwares are already compatible with ARM64, including the k3s we are going to install in this tutorial.
You might have noticed in line 7 that we are passing in a cloud-init5 file. That's an important piece of the whole puzzle. Let's create that file.
Generating SSH key pair
If you don't have an SSH key pair, you can generate one using the following command:
#cloud-config
users:
- name: k8s
groups: users, admin
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- # Place the hardcoded value of your SSH public key here
# Output of `ssh-keygen` .pub file
packages:
- fail2ban
- python3
- python3-pip
package_update: true
package_upgrade: true
runcmd:
# Allow reading the logs using journalctl
- usermod -aG adm k8s
# Ban malicious IPs from overly aggressive SSH login attempts
- printf "[sshd]\nenabled = true\nbanaction = iptables-multiport" > /etc/fail2ban/jail.local
- systemctl enable fail2ban
# Configure SSH daemon in a more secure way
- sed -i -e '/^\(#\|\)PermitRootLogin/s/^.*$/PermitRootLogin no/' /etc/ssh/sshd_config
- sed -i -e '/^\(#\|\)PasswordAuthentication/s/^.*$/PasswordAuthentication no/' /etc/ssh/sshd_config
- sed -i -e '/^\(#\|\)KbdInteractiveAuthentication/s/^.*$/KbdInteractiveAuthentication no/' /etc/ssh/sshd_config
- sed -i -e '/^\(#\|\)ChallengeResponseAuthentication/s/^.*$/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
- sed -i -e '/^\(#\|\)MaxAuthTries/s/^.*$/MaxAuthTries 2/' /etc/ssh/sshd_config
- sed -i -e '/^\(#\|\)AllowTcpForwarding/s/^.*$/AllowTcpForwarding no/' /etc/ssh/sshd_config
- sed -i -e '/^\(#\|\)X11Forwarding/s/^.*$/X11Forwarding no/' /etc/ssh/sshd_config
- sed -i -e '/^\(#\|\)AllowAgentForwarding/s/^.*$/AllowAgentForwarding no/' /etc/ssh/sshd_config
- sed -i -e '/^\(#\|\)AuthorizedKeysFile/s/^.*$/AuthorizedKeysFile .ssh\/authorized_keys/' /etc/ssh/sshd_config
- sed -i '$a AllowUsers k8s' /etc/ssh/sshd_config
# Install the Kubernetes cluster using k3s
- |
curl https://get.k3s.io | \
INSTALL_K3S_VERSION="v1.29.2+k3s1" \
INSTALL_K3S_EXEC="--disable traefik
--disable-network-policy
--flannel-backend none
--write-kubeconfig /home/k8s/.kube/config
--secrets-encryption" \
sh -
- chown -R k8s:k8s /home/k8s/.kube/
# Ensure all the settings are applied
- reboot
The first line of this file is not a normal comment. It's a directive that tells the cloud-init to ignore the rest of the file if it's not a valid cloud-init config. It has a similar behavior to the shebang in shell scripts.
Shebang
The shebang is a special comment that tells the shell which interpreter to use to execute the script. It's usually the first line of a script file.
As soon as I make this script executable using chmod +x hello.sh
, I can run it using ./hello.sh
and by parsing the first line, the shell knows that it should use the bash
interpreter to execute the script.
Explain the k3s installation¶
All the config in cloud-init file is self-explanatory. However, I'd like to highlight a few important points.
Config/Flag | Description |
---|---|
--disable-network-policy | This flag disables the default network policy controller in k3s. We will use Cilium as the CNI plugin, which comes with its own network policy controller. |
--flannel-backend none | This flag disables the default Flannel CNI plugin in k3s. Again, we will use Cilium as the CNI plugin. |
--secrets-encryption | This flag enables the secrets encryption feature in k3s. This is a security feature that encrypts the secrets stored in etcd. |
--disable traefik | We will not require Traefik as an Ingress Controller in our setup. We will use Cilium with the help of ServiceLB of k3s. |
Finally, let's create the relevant output that we'll use for later steps.
output "server_ipv4_address" {
value = hcloud_server.this.ipv4_address
}
output "server_ipv6_address" {
value = hcloud_server.this.ipv6_address
}
We have our TF files ready for provisioning the server. Let's create the server using the tofu
CLI.
cd opentofu
tofu init
tofu plan -out tfplan
# observe the plan visually and confirm the changes
tofu apply tfplan
Now that we've got the IP address, we can use either the IPv4 or the IPv6 to connect to the server.
IP_ADDRESS=$(tofu output -raw server_ipv4_address)
echo "${IP_ADDRESS} k3s-cluster.hetzner" | sudo tee -a /etc/hosts
This will ensure that the hostname k3s-cluster.hetzner
resolves to the IP address of the server and we won't have to memorize the IP address for every command we run.
Step 1.1: Prepare Ansible Inventory¶
We will heavily rely on the SSH connections to the machine as the default way of connection for Ansible. Therefore, to make that access easier and password-less, we gotta take two steps:
- Prepare the SSH config file
- Prepare the Ansible inventory file
Let's start with the SSH config file.
# ... truncated ...
Host k3s-cluster.hetzner
User k8s
IdentityFile ~/.ssh/k3s-cluster.hetzner # location of `ssh-keygen` command
Now, let's prepare the Ansible inventory file using the yaml
plugin of Ansible inventory.
Step 2: Install Kubernetes on Ubuntu 22.04¶
So far, we have provisioned the cluster using OpenTofu. But, if you SSH into the machine, kubectl get nodes
will return a NotReady
state for your node. That is because of the absence of a CNI plugin in the cluster.
It's time to use Ansible to take care of that, although, arguably, we could have done that in the cloud-init file as well.
But, using Ansible gives more flexibility and control over the steps involved and it will make it reproducible upon future invocations.
- name: Bootstrap the Kubernetes cluster
hosts: all
gather_facts: true
become: true
vars:
helm_version: v3.14.3
tasks:
- name: Install Kubernetes library
ansible.builtin.pip:
name: kubernetes<30
state: present
- name: Install helm binary
ansible.builtin.shell:
cmd: "{{ lookup('ansible.builtin.url', 'https://git.io/get_helm.sh', split_lines=false) }}"
creates: /usr/local/bin/helm
environment:
DESIRED_VERSION: "{{ helm_version }}"
- name: Install Kubernetes gateway CRDs
kubernetes.core.k8s:
src: "{{ item }}"
state: present
loop:
- https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
- https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
- name: Install cilium
block:
- name: Add cilium helm repository
kubernetes.core.helm_repository:
name: cilium
repo_url: https://helm.cilium.io
- name: Install cilium helm release
kubernetes.core.helm:
name: cilium
chart_ref: cilium/cilium
namespace: kube-system
state: present
chart_version: 1.15.2
values:
gatewayAPI:
enabled: true
kubeProxyReplacement: true
encryption:
enabled: true
type: wireguard
operator:
replicas: 1
As you see in the highlighted lines, we will use Kubernetes Gateway API6 as a replacement for Ingress Controller. This is a more modern approach and it has more capabilities than the traditional Ingress Controller.
Among many other benefits, Gateway API has the ability to handle TLS, gRPC, and WebSockets, which are not supported by Ingress Controller.
Why Gateway API?
These days I will use Gateway API for all my Kubernetes clusters, unless I have a specific requirement that mandates the use of Ingress Controller.
Cilium has native support7 for Gatway API.
To run this ansible playbook, I will simply run the following command:
You might as well ignore that last -v
flag, but I like to see the output of the playbook as it runs.
Ansible Configuration
In fact, I have a global user configuration that has some sane defaults. Completely contradictory to what the Ansible sane defaults are.
[defaults]
become=false
log_path=/tmp/ansible.log
gather_facts=false
fact_caching = ansible.builtin.jsonfile
fact_caching_connection = /tmp/ansible_facts
cache_timeout = 3600
interpreter_python = auto_silent
verbosity = 2
ssh_common_args = -o ConnectTimeout=5
[inventory]
enable_plugins = 'host_list', 'script', 'auto', 'yaml', 'ini', 'toml', 'azure_rm', 'aws_ec2', 'auto'
cache = yes
cache_connection = /tmp/ansible_inventory
Step 3: K3s Ubuntu¶
We have done all the heavy lifting thus far and now it's time to use the cluster.
For that, you can either use scp
or rsync
to fetch the ~/.kube/config
into your local machine, or use the kubectl
command right from the remote server.
I generally prefer the former as I believe the control machine is my localhost and everything else is a remote machine that hosts the workload I command it to.
NOTE: In the case where you copy the remote Kubeconfig file to your local machine, you will need to update the server
field in the ~/.kube/config
file as it is pointing to the 127.0.0.1
by default.
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://127.0.0.1:6443 # CHANGE THIS LINE
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
client-certificate-data: DATA+OMITTED
client-key-data: DATA+OMITTED
Step 4: Protect the Server¶
Since exposing the Kubernetes API server means the cluster shall be internet-accessible, I, as a security-conscious person, would protect my server with the help of firewall.
That said, I would, as a last step, deploy the free Hetzner Cloud Firewall to protect my server.
terraform {
required_providers {
hcloud = {
source = "hetznercloud/hcloud"
version = "~> 1.45"
}
http = {
source = "hashicorp/http"
version = "~> 3.4"
}
}
required_version = "<2"
}
variable "hetzner_api_token" {
type = string
nullable = false
sensitive = true
}
provider "hcloud" {
token = var.hetzner_api_token
}
data "http" "this" {
url = "https://api.ipify.org"
}
resource "hcloud_firewall" "this" {
name = "k3s-cluster"
dynamic "rule" {
for_each = toset({
ssh = 22
kubernetes_api = 6443
})
content {
direction = "in"
protocol = "tcp"
port = rule.value
source_ips = [
format("%s/32", data.http.this.response_body),
]
description = "Admin public IP address"
}
}
}
resource "hcloud_firewall_attachment" "this" {
firewall_id = hcloud_firewall.this.id
server_ids = [hcloud_server.this.id]
}
Bonus: Multi-Node Cluster¶
When planning to go for production in this setup, you are advised to go for a multi-node deployment for high availability and fault tolerance.
To achieve that, you can pass the K3S_URL
8 and K3S_TOKEN
to the cloud-init script for any of the worker nodes.
Source Code¶
All the code examples in this post are publicly available9 on GitHub under the Apache 2.0 license.
If you enjoyed this blog post, consider sharing it with these buttons . Please leave a comment for us at the end, we read & love 'em all.
Share on Share on Share on Share on
-
https://community.hetzner.com/tutorials/setup-your-own-scalable-kubernetes-cluster ↩
-
https://docs.hetzner.com/cloud/api/getting-started/generating-api-token/ ↩
-
https://github.com/rust-lang/simpleinfra/blob/e361f222bc377434d06d53add6827bd24a3a5d89/ansible/apply#L13:L15 ↩
-
https://cloudinit.readthedocs.io/en/latest/topics/format.html ↩
-
https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/ ↩
-
https://github.com/developer-friendly/blog/tree/main/docs/blog/codes/2024/0005-install-k3s-ubuntu ↩