Sealed Secrets on K3s (Vagrant/VirtualBox): A Practical, Step-by-Step Guide with Troubleshooting

DevOps, Homelab For Devops, Linux

In this tutorial you’ll deploy Bitnami Sealed Secrets on a three-node K3s cluster running in Vagrant/VirtualBox, generate the public key, and create your first encrypted Secret. Every command is explained so you can teach it or repeat it later without guesswork.

Environment & Assumptions

  • Kubernetes: K3s (single control plane)
  • VMs: Vagrant + VirtualBox
  • Nodes:
nodes = [ 
  { name: "k3s-master", hostname: "k3s-master", public_ip: "192.168.0.101", private_ip: "192.168.230.10", mem: 3072, cpus: 2 }, 
  { name: "k3s-worker-1", hostname: "k3s-worker-1", public_ip: "192.168.0.102", private_ip: "192.168.230.11", mem: 2048, cpus: 2 }, 
  { name: "k3s-worker-2", hostname: "k3s-worker-2", public_ip: "192.168.0.103", private_ip: "192.168.230.12", mem: 2048, cpus: 2 }, 
]

You will run commands as the vagrant user on the master node unless noted.

What Are Sealed Secrets (in one paragraph)

Sealed Secrets let you commit encrypted Kubernetes Secrets to Git safely. A cluster-side controller (the sealed-secrets-controller) decrypts them into native Secrets only inside your cluster. You encrypt locally with kubeseal using the controller’s public key; only the controller’s private key can decrypt.

1) Install the kubeseal CLI on the VM

# Download & install kubeseal v0.27.0 
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/kubeseal-0.27.0-linux-amd64.tar.gz 

tar -xvzf kubeseal-0.27.0-linux-amd64.tar.gz 

sudo install -m 755 kubeseal /usr/local/bin/kubeseal kubeseal --version
  • wget … downloads the tarball containing the kubeseal binary.
  • tar -xvzf extracts it; flags: x (extract), v (verbose), z (gzip), f (file).
  • install … moves the binary into your $PATH with executable permissions.
  • kubeseal –version confirms it’s usable.

2) Deploy the Sealed Secrets Controller

Important: Bitnami’s default manifest installs into the kube-system namespace. Don’t override the namespace on the CLI, or you’ll get “namespace … does not match” errors.

# Apply the upstream controller manifest AS-IS (installs to kube-system) 

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/controller.yaml

What this does:

  • Creates the CRD sealedsecrets.bitnami.com so the API server knows about SealedSecret objects.
  • Creates RBAC, ServiceAccount, Services, and the sealed-secrets-controller Deployment in kube-system.

Verify it’s running:

# List Deployments in kube-system, filter for sealed-secrets 

kubectl get deploy -n kube-system | grep sealed-secrets
kubectl get pods -n kube-system | grep sealed-secrets
  • If you see “No resources found” but the Deployment exists, the Pod may still be starting or a label selector was wrong. Use a broad listing (kubectl get all -n kube-system | grep -i sealed) to confirm.

3) Fix Kubeconfig for the vagrant User

On K3s, the admin kubeconfig lives at /etc/rancher/k3s/k3s.yaml (owned by root). To make kubectl/kubeseal work for the vagrant user without setting KUBECONFIG each time, copy it.

# One-time setup for the vagrant user 

mkdir -p ~/.kube 
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config 
sudo chown "$(id -u)":"$(id -g)" ~/.kube/config
  • mkdir -p creates the kube config directory if missing.
  • cp copies the cluster’s admin kubeconfig.
  • chown fixes file ownership so kubectl can read it.

Temporary alternative: set KUBECONFIG inline for the current shell:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

4) Export the Controller’s Public Certificate

Now that your kubeconfig is set, fetch the public key that kubeseal uses to encrypt.

# Save the controller's public cert (used by kubeseal for encryption) 

kubeseal \
 --fetch-cert \
 --controller-namespace=kube-system \
 --controller-name=sealed-secrets-controller \
 > ~/pub-sealed-secrets.pem 

# Inspect it (PEM formatted X.509 cert) 
head -n 5 ~/pub-sealed-secrets.pem
  • –fetch-cert asks the controller for its public certificate.
  • –controller-namespace/name must match where you installed it (default is kube-system and name sealed-secrets-controller).

Common error: invalid configuration: no configuration has been provided → your kubeconfig isn’t set; fix it with the steps in Section 3.

5) Create & Seal Your First Secret

Start with a normal Secret manifest generated client-side:

# Create a plaintext Secret manifest WITHOUT applying to the cluster 

kubectl create secret generic my-secret \
 --namespace=default \
 --from-literal=password=SuperSecretPassword2025 \
 --dry-run=client -o yaml > plain.yaml
  • –dry-run=client -o yaml prints the manifest instead of sending it to the API. This avoids storing any secret in etcd.

Seal it with the controller’s public cert:

# Convert the plain Secret to a SealedSecret (safe for Git) 

kubeseal --cert ~/pub-sealed-secrets.pem --format=yaml < plain.yaml > sealed.yaml
  • –cert tells kubeseal to use your saved public key (no cluster access required during sealing).
  • –format=yaml outputs a SealedSecret custom resource.

Apply the sealed resource to the cluster:

kubectl apply -f sealed.yaml

The controller will create a native Secret:

# Confirm the decrypted Secret exists 

kubectl get secret my-secret -n default

# Read and decode the value (demo only)

kubectl get secret my-secret -n default -o jsonpath='{.data.password}' | base64 -d

Troubleshooting (Real-World Gotchas)

“No resources found in sealed-secrets namespace.”

You applied the manifest with -n sealed-secrets. The upstream file is built for kube-system. Fix by deleting the wrong namespace (if created), and apply the manifest without overriding the namespace:

kubectl delete namespace sealed-secrets --ignore-not-found 
kubectl delete -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/controller.yaml --ignore-not-found 
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/controller.yaml

“invalid configuration: no configuration has been provided”

Your shell can’t find a kubeconfig. Either export it temporarily:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Or copy it permanently for the vagrant user as shown in Section 3.

You can see the Deployment but not the Pod

Give it a minute, then list everything broadly to avoid label selector mismatches:

kubectl get all -n kube-system | grep -i sealed kubectl describe deploy sealed-secrets-controller -n kube-system kubectl describe pod -l app.kubernetes.io/name=sealed-secrets -n kube-system

SealedSecret applied, but no Secret appears

  • Check controller logs: kubectl logs -n kube-system deploy/sealed-secrets-controller
  • Make sure the metadata.namespace in your SealedSecret matches where you expect the plain Secret to be created.

Operational Tips

Where to commit what

  • Commit to Git: sealed.yaml (the SealedSecret)
  • Never commit: plain.yaml or any unsealed Secrets

Regenerating the public certificate

kubeseal --fetch-cert \
 --controller-namespace=kube-system \
 --controller-name=sealed-secrets-controller \
 > ~/pub-sealed-secrets.pem

Key rotation (high level)

  1. Enable key renewal in the controller (check the controller flags/ConfigMap).
  2. Fetch the new public cert and re-seal Secrets only if you rotate keys with breaking changes. The controller supports decrypting with old private keys until you prune them.

Using a different namespace

You can seal for any namespace/name pair. Ensure your plain.yaml has the correct metadata.namespace and metadata.name before sealing, because kubeseal binds encryption to those values.

Full Command Recap (Copy-Paste Ready)

# 0) (Optional) On k3s-master, prepare kubeconfig for the vagrant user 

mkdir -p ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config 
sudo chown "$(id -u)":"$(id -g)" ~/.kube/config 

# 1) Install 

kubeseal wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/kubeseal-0.27.0-linux-amd64.tar.gz tar -xvzf kubeseal-0.27.0-linux-amd64.tar.gz 
sudo install -m 755 kubeseal /usr/local/bin/kubeseal 

# 2) Deploy controller into kube-system (default from upstream) 
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/controller.yaml 
kubectl get deploy -n kube-system | grep sealed-secrets 
kubectl get pods -n kube-system | grep sealed-secrets 

# 3) Export public cert 
kubeseal --fetch-cert --controller-namespace=kube-system --controller-name=sealed-secrets-controller > ~/pub-sealed-secrets.pem 

# 4) Create and seal a Secret 
kubectl create secret generic my-secret \ --namespace=default \ --from-literal=password=SuperSecretPassword2025 \ --dry-run=client -o yaml > plain.yaml 
kubeseal --cert ~/pub-sealed-secrets.pem --format=yaml < plain.yaml > sealed.yaml 
kubectl apply -f sealed.yaml 

# 5) Verify 
kubectl get secret my-secret -n default -o jsonpath='{.data.password}' | base64 -d; echo

FAQ

Q: Can I seal on my laptop and apply in CI?
A: Yes. As long as you use the target cluster’s public cert (or a certificate from a cluster with the same private key), sealing is offline. CI/CD just applies SealedSecret manifests.

Q: What if I re-install my cluster?
A: The controller’s private key changes by default, so previously sealed manifests won’t decrypt. Backup the controller’s private key Secret (in kube-system) to persist across rebuilds, or be ready to re-seal with the new public cert.

Q: Is it safe to print Secret values like we did?
A: Only for demos. In real pipelines avoid printing decoded values. Use mounting/env injection directly from the Secret.

Conclusion

You’ve installed the Sealed Secrets controller on K3s, fixed kubeconfig for the vagrant user, exported the public cert, and successfully sealed/applied a Secret. With this in place, you can store all sensitive app configuration in Git—encrypted end-to-end—while your cluster handles secure decryption at deploy time.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.