Installing K3s + K3d and Deploying a K3d Kubernetes Cluster in Docker (with a fully persistent volumes)

DevOps, Docker, Homelab For Devops

If you want “real Kubernetes behavior” but don’t want to burn a whole fleet of VMs, K3d is one of the fastest ways to spin up a Kubernetes lab. It runs K3s (a lightweight Kubernetes distribution) inside Docker containers, so you can create and destroy clusters in seconds, while still keeping data persistent on the host.

In this guide we’ll:

  • Explain what K3s vs K3d is (and when you use each)
  • Install K3s (optional, but useful to know) and install K3d
  • Deploy a K3d cluster using a YAML config
  • Persist all Kubernetes data under /srv/data/k3d
  • Walk through your lab.yaml line by line

K3s vs K3d: what’s the difference?

  • K3s is a lightweight Kubernetes distribution. You typically install it on a Linux host (or VM) and it runs as a system service.
  • K3d is a wrapper tool that runs K3s inside Docker containers. It’s built for labs, CI, quick testing, and “spin up / tear down” workflows.

In practice:

  • Use K3s when you want a small, efficient Kubernetes on servers/VMs (edge, lab servers, small prod, etc.).
  • Use K3d when you want Kubernetes fast, isolated, and disposable on a Docker host.

Your /srv layout (recommended for lab hygiene)

You already have a clean structure. This is exactly the kind of layout that keeps labs maintainable:

/srv/
├── containers
│   └── k3d
│       └── cluster
│           └── lab.yaml
├── data
│   └── k3d
│       ├── agent0
│       ├── agent1
│       ├── kubeconfig
│       ├── server
│       └── storage
└── log
    └── k3d

Rule of thumb:

  • /srv/containers → configs / manifests / compose files
  • /srv/data → persistent state (databases, k8s data dirs, etc.)
  • /srv/log → host-side logs you want to keep

Prerequisites

  • Linux host (Debian/Ubuntu/RHEL-based — commands below are generic)
  • Docker installed and running
  • curl, openssl
  • kubectl on the host (recommended; you can also use the kubectl bundled inside K3d/K3s workflows, but host kubectl is simpler)

Quick sanity checks:

docker version
docker info
kubectl version --client

(Optional) Install K3s on the host

You do not need K3s installed on the host to use K3d — K3d brings its own K3s inside containers. But many admins like to install K3s once just to understand the “native” layout and tooling.

# Install K3s (server mode)
curl -sfL https://get.k3s.io | sh -

# kubeconfig for root is typically:
# /etc/rancher/k3s/k3s.yaml
sudo kubectl get nodes

If you’re only doing K3d labs on this host, you can skip this entire section.

Install K3d (recommended method)

Install K3d via the official install script:

curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
k3d version

(If you prefer pinned versions for repeatability, you can set TAG=vX.Y.Z before running the script.)

Prepare directories

Make sure everything exists and has sane permissions. (Use root-owned directories if you run k3d as root; therwise ensure your user can write there.)

sudo mkdir -p /srv/containers/k3d/cluster
sudo mkdir -p /srv/data/k3d/{server,agent0,agent1,storage,kubeconfig}
sudo mkdir -p /srv/log/k3d

# Optional: set ownership if you run as a non-root user
# sudo chown -R $USER:$USER /srv/containers /srv/data /srv/log

Generate a cluster token (shared secret)

K3s nodes use a token to join the cluster. In K3d, you can pass it via K3s args. Generate one:

openssl rand -hex 32

You’ll paste that value into lab.yaml as --token=YOURTOKEN.
Treat it like a password.

Create the Docker network (matches your YAML)

Your config expects a Docker network called net-k3s:

docker network create net-k3s

If it already exists, Docker will tell you. That’s fine.

Your K3d cluster config (lab.yaml)

Here is your config as provided (token redacted). I’ll explain every line right after.

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: lab

servers: 1
agents: 2

network: net-k3s

options:
  k3d:
    wait: true
    timeout: "180s"
    disableLoadbalancer: true
  k3s:
    extraArgs:
      - arg: "--token=..."
        nodeFilters:
          - server:*
      - arg: "--token=..."
        nodeFilters:
          - agent:*

volumes:
  - volume: /srv/data/k3d/server:/var/lib/rancher/k3s/server
    nodeFilters:
      - server:0
  - volume: /srv/data/k3d/agent0:/var/lib/rancher/k3s
    nodeFilters:
      - agent:0
  - volume: /srv/data/k3d/agent1:/var/lib/rancher/k3s
    nodeFilters:
      - agent:1
  - volume: /srv/data/k3d/storage:/var/lib/rancher/k3s/storage
    nodeFilters:
      - all

lab.yaml explained line by line (what each field does)

API and object type

  • apiVersion: k3d.io/v1alpha5
    The schema version for K3d’s config format. This tells K3d how to parse the YAML.
  • kind: Simple
    A “simple” cluster definition (as opposed to more advanced/custom kinds). Good for labs.

Metadata

  • metadata:
    Standard Kubernetes-style metadata block.
  • name: lab
    The cluster name becomes part of container names, kubeconfig contexts, etc. Your cluster will be called lab.

Cluster size

  • servers: 1
    Number of K3s server nodes (control-plane). In K3d these are Docker containers (e.g. k3d-lab-server-0).
  • agents: 2
    Number of K3s agent nodes (workers). Also Docker containers (e.g. k3d-lab-agent-0, k3d-lab-agent-1).

This gives you a realistic topology: 1 control-plane + 2 workers.

Networking

  • network: net-k3s
    Tells K3d to attach the cluster containers to an existing Docker network named net-k3s. This is handy if you want other containers to talk to the cluster on a shared lab network.

Options → k3d

  • options:
    A block for tool-specific options.
  • k3d:
    Options that control K3d’s behavior (not Kubernetes itself).
  • wait: true
    K3d will wait until the cluster is ready before returning success. Helpful for automation/scripts.
  • timeout: "180s"
    Maximum time to wait for readiness (here: 180 seconds). Quoted because it’s a duration string.
  • disableLoadbalancer: true
    K3d normally creates a “server load balancer” container for exposing services/ports more conveniently. You’re disabling it, which is totally valid for labs where you prefer explicit port mappings or ingress control later.

Options → k3s

  • k3s:
    Options passed down to K3s (the Kubernetes distribution running inside the containers).
  • extraArgs:
    A list of extra command-line arguments that K3d will add when starting K3s server/agent processes.

Now the important part:

  • - arg: "--token=..."
    This injects the cluster token. In K3s, servers and agents need to share a token so agents can join the cluster.
  • nodeFilters:
    Controls which nodes receive that argument.
  • - server:*
    Apply this arg to all server nodes (server-0, server-1, … if you had more).
  • The second block repeats --token but targets the agents:
  • - agent:* applies the token to all agent nodes.

Why do it twice? Because you’re being explicit: servers get the token, and agents get the token. That’s clean and readable.

Volumes (persistent storage)

This section is the heart of making your K3d cluster persistent across rebuilds. Each entry maps a host directory to a container path.

  • volumes:
    A list of volume mappings.

1) Server data

  • volume: /srv/data/k3d/server:/var/lib/rancher/k3s/server
    Maps the K3s server’s main state directory to the host. This is where the server stores the datastore (SQLite by default, or etcd in HA setups), manifests, certs, etc.
  • nodeFilters: - server:0
    Apply this volume mapping only to the first server node container.

2) Agent 0 data

  • volume: /srv/data/k3d/agent0:/var/lib/rancher/k3s
    Persists the agent’s K3s runtime state (agent-specific data).
  • nodeFilters: - agent:0
    Apply only to worker node 0.

3) Agent 1 data

  • volume: /srv/data/k3d/agent1:/var/lib/rancher/k3s
    Same concept, for worker node 1.
  • nodeFilters: - agent:1
    Apply only to worker node 1.

4) Shared storage directory

  • volume: /srv/data/k3d/storage:/var/lib/rancher/k3s/storage
    A shared path available on all nodes. This is useful for lab experiments (e.g., simple hostPath-ish workflows, shared files, or testing local provisioners).
  • nodeFilters: - all
    Attach it to every server and agent container.

⚠️ Note: “shared storage” here means “shared host directory mounted into multiple containers”, not a real distributed storage system. It’s perfect for labs, but for production you’d look at Longhorn, Rook/Ceph, NFS, etc.

Create the cluster from YAML

k3d cluster create --config /srv/containers/k3d/cluster/lab.yaml

Because you set wait: true, K3d should return only after the cluster is ready (or after 180 seconds).

Get kubeconfig and make kubectl talk to the cluster

K3d can merge kubeconfig into your default ~/.kube/config, but you’re keeping things under /srv/data/k3d/kubeconfig (nice for server-side labs).

# Write kubeconfig for this cluster to a dedicated file
k3d kubeconfig get lab > /srv/data/k3d/kubeconfig/lab.yaml

# Use it for kubectl in this shell
export KUBECONFIG=/srv/data/k3d/kubeconfig/lab.yaml

kubectl get nodes
kubectl get pods -A

If you want it persistent for your user, you can add the export to your shell profile.

Validate everything (quick checklist)

# Cluster containers
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Nodes
kubectl get nodes -o wide

# System pods
kubectl -n kube-system get pods -o wide

# Confirm your host volumes are being used
ls -la /srv/data/k3d/server
ls -la /srv/data/k3d/agent0
ls -la /srv/data/k3d/agent1

Common tweaks (practical lab improvements)

1) Add a built-in load balancer (if you want easy port exposure)

Right now you disabled it:

disableLoadbalancer: true

If later you want “easy mode” service exposure, set it to false (or remove the line), recreate the cluster, and use the LB container.

2) Add explicit port mappings (great for Ingress experiments)

You can expose ports from the LB or server container depending on your setup. With LB disabled, you’d  typically map ports directly (not shown in your YAML yet). If you decide to add this later, do it intentionally (to avoid port collisions on a shared lab host).

3) Keep logs readable

You already have /srv/log/k3d. For runtime troubleshooting, these are gold:

# K3d cluster info
k3d cluster list
k3d node list

# Container logs (example)
docker logs k3d-lab-server-0 --tail 200
docker logs k3d-lab-agent-0 --tail 200

Troubleshooting (fast fixes)

Docker network missing

If cluster creation fails with a network error:

docker network ls | grep net-k3s
docker network create net-k3s

Token mismatch / nodes not joining

Make sure the token is identical for server and agent args. If you rotated it, recreate the cluster cleanly:

k3d cluster delete lab
k3d cluster create --config /srv/containers/k3d/cluster/lab.yaml

Dirty persistent data after experiments

Because you persist state under /srv/data/k3d, “bad lab days” can survive rebuilds (that’s both a feature and a curse). To fully reset:

k3d cluster delete lab
sudo rm -rf /srv/data/k3d/server/* /srv/data/k3d/agent0/* /srv/data/k3d/agent1/*

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.