HomeLab – Sealed Secrets on k3s (Helm install, kubeseal fix, and end-to-end verification) – Day 12

DevOps, Homelab For Devops, Linux

Today we rolled out Bitnami Sealed Secrets to our k3s cluster using Helm, hit a snag with the kubeseal CLI artifact, fixed it by pinning a working release, and verified the full round-trip: SealedSecret → controller decrypts → Secret recreated after deletion. To keep the day lean, we used a minimal number of Ansible playbooks and made them idempotent.

What we built

  • Installed the Sealed Secrets controller via Helm in kube-system.
  • Installed a working kubeseal binary on the control node.
  • Created a demo SealedSecret for apps/demo-secret.
  • Verified the controller decrypts and (re)creates the underlying Secret.

Prerequisites

  • k3s control node reachable over SSH (k3s-master in inventory).
  • KUBECONFIG on the node: /etc/rancher/k3s/k3s.yaml.
  • Helm present on the node.

Playbook 1 — day12_sealed_secrets.yml

Installs the controller via Helm, installs a working kubeseal, and seals a demo Secret.

---
- name: Install Sealed Secrets controller (Helm) on cluster
  hosts: k3s-master
  gather_facts: false
  vars:
    kubeconfig_path: /etc/rancher/k3s/k3s.yaml
    controller_namespace: kube-system
    controller_name: sealed-secrets
  tasks:
    - name: Add Helm repo (idempotent)
      shell: |
        set -euo pipefail
        helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets 2>&1 || true
        echo "ok"
      args:
        executable: /bin/bash
      changed_when: "'already exists' not in (result.stdout | default(''))"
      register: result

    - name: Helm repo update
      command: helm repo update
      changed_when: false

    - name: Install/upgrade Sealed Secrets controller
      command: >
        helm upgrade --install {{ controller_name }} sealed-secrets/sealed-secrets
        -n {{ controller_namespace }} --create-namespace
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Wait for controller rollout
      command: >
        kubectl -n {{ controller_namespace }} rollout status deploy/{{ controller_name }} --timeout=180s
      changed_when: false
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

- name: Install kubeseal and create a demo SealedSecret
  hosts: k3s-master
  gather_facts: false
  vars:
    kubeconfig_path: /etc/rancher/k3s/k3s.yaml

    # Working version (tar layout includes a "kubeseal" binary)
    kubeseal_version: v0.24.5
    kubeseal_url: "https://github.com/bitnami-labs/sealed-secrets/releases/download/{{ kubeseal_version }}/kubeseal-{{ kubeseal_version }}-linux-amd64.tar.gz"
    kubeseal_tar: /tmp/kubeseal.tar.gz
    kubeseal_bin: /usr/local/bin/kubeseal

    controller_namespace: kube-system
    controller_name: sealed-secrets
    controller_cert: /tmp/sealed-secrets.pem

    demo_ns: apps
    demo_secret_name: demo-secret
    demo_secret_manifest: "/tmp/{{ demo_secret_name }}.yaml"
    demo_sealed_manifest: "/tmp/{{ demo_secret_name }}-sealed.yaml"
    demo_username: demo
    demo_password: "S3cretP@ssw0rd!"
  tasks:
    - name: Check current kubeseal works
      command: "{{ kubeseal_bin }} --version"
      register: kubeseal_ver
      failed_when: false
      changed_when: false

    - name: Remove broken kubeseal (if needed)
      file:
        path: "{{ kubeseal_bin }}"
        state: absent
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Download kubeseal tarball (fresh)
      get_url:
        url: "{{ kubeseal_url }}"
        dest: "{{ kubeseal_tar }}"
        mode: '0644'
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Extract kubeseal
      unarchive:
        src: "{{ kubeseal_tar }}"
        dest: /tmp
        remote_src: true
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Install kubeseal to /usr/local/bin
      copy:
        src: /tmp/kubeseal
        dest: "{{ kubeseal_bin }}"
        mode: '0755'
        remote_src: true
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Verify kubeseal installed
      command: "{{ kubeseal_bin }} --version"
      changed_when: false

    - name: Fetch controller public cert (PEM)
      shell: |
        set -euo pipefail
        {{ kubeseal_bin }} \
          --controller-name {{ controller_name }} \
          --controller-namespace {{ controller_namespace }} \
          --fetch-cert > {{ controller_cert }}
      args:
        executable: /bin/bash
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Ensure namespace exists for demo
      command: "kubectl get ns {{ demo_ns }}"
      register: ns_check
      failed_when: false
      changed_when: false
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Create namespace if missing
      command: "kubectl create ns {{ demo_ns }}"
      when: ns_check.rc != 0
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Render demo Secret manifest (unencrypted, local temp)
      copy:
        dest: "{{ demo_secret_manifest }}"
        mode: '0600'
        content: |
          apiVersion: v1
          kind: Secret
          metadata:
            name: {{ demo_secret_name }}
            namespace: {{ demo_ns }}
          type: Opaque
          stringData:
            username: "{{ demo_username }}"
            password: "{{ demo_password }}"

    - name: Seal the Secret into a SealedSecret
      shell: |
        set -euo pipefail
        cat {{ demo_secret_manifest }} \
        | {{ kubeseal_bin }} --format=yaml --cert {{ controller_cert }} \
        > {{ demo_sealed_manifest }}
      args:
        executable: /bin/bash
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Apply SealedSecret (idempotent)
      command: "kubectl apply -f {{ demo_sealed_manifest }}"
      register: apply_sealed
      changed_when: "'created' in (apply_sealed.stdout|default('')) or 'configured' in (apply_sealed.stdout|default(''))"
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Wait for decrypted Secret to appear
      shell: |
        set -euo pipefail
        kubectl -n {{ demo_ns }} get secret {{ demo_secret_name }} -o name
      args:
        executable: /bin/bash
      register: have_secret
      retries: 30
      delay: 3
      until: have_secret.rc == 0
      changed_when: false
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

Playbook 2 — day12_sealed_secrets-fix.yml

Focused “fix” when the first kubeseal attempt failed due to an incompatible tarball. Pins v0.24.5 and repeats the sealing flow.

---
- name: Fix kubeseal and seal demo secret
  hosts: k3s-master
  gather_facts: false
  vars:
    kubeconfig_path: /etc/rancher/k3s/k3s.yaml
    kubeseal_version: v0.32.2
    kubeseal_tar_version: 0.32.2
    kubeseal_url: "https://github.com/bitnami-labs/sealed-secrets/releases/download/{{ kubeseal_version }}/kubeseal-{{ kubeseal_tar_version }}-linux-amd64.tar.gz"
    kubeseal_tar: /tmp/kubeseal.tar.gz
    kubeseal_bin: /usr/local/bin/kubeseal

    controller_namespace: kube-system
    controller_name: sealed-secrets
    controller_cert: /tmp/sealed-secrets.pem

    demo_ns: apps
    demo_secret_name: demo-secret
    demo_secret_manifest: "/tmp/{{ demo_secret_name }}.yaml"
    demo_sealed_manifest: "/tmp/{{ demo_secret_name }}-sealed.yaml"
    demo_username: demo
    demo_password: "S3cretP@ssw0rd!"

  tasks:
    - name: Check current kubeseal works
      command: "{{ kubeseal_bin }} --version"
      register: kubeseal_ver
      failed_when: false
      changed_when: false

    - name: Remove broken kubeseal (if needed)
      file:
        path: "{{ kubeseal_bin }}"
        state: absent
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Download kubeseal tarball (fresh)
      get_url:
        url: "{{ kubeseal_url }}"
        dest: "{{ kubeseal_tar }}"
        mode: '0644'
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Extract kubeseal
      unarchive:
        src: "{{ kubeseal_tar }}"
        dest: /tmp
        remote_src: true
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Install kubeseal to /usr/local/bin
      copy:
        src: /tmp/kubeseal
        dest: "{{ kubeseal_bin }}"
        mode: '0755'
        remote_src: true
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Verify kubeseal installed
      command: "{{ kubeseal_bin }} --version"
      changed_when: false

    - name: Fetch controller public cert (PEM)
      shell: |
        set -euo pipefail
        {{ kubeseal_bin }} \
          --controller-name {{ controller_name }} \
          --controller-namespace {{ controller_namespace }} \
          --fetch-cert > {{ controller_cert }}
      args:
        executable: /bin/bash
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Ensure namespace exists for demo
      command: "kubectl get ns {{ demo_ns }}"
      register: ns_check
      failed_when: false
      changed_when: false
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Create namespace if missing
      command: "kubectl create ns {{ demo_ns }}"
      when: ns_check.rc != 0
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Render demo Secret manifest (unencrypted, local temp)
      copy:
        dest: "{{ demo_secret_manifest }}"
        mode: '0600'
        content: |
          apiVersion: v1
          kind: Secret
          metadata:
            name: {{ demo_secret_name }}
            namespace: {{ demo_ns }}
          type: Opaque
          stringData:
            username: "{{ demo_username }}"
            password: "{{ demo_password }}"

    - name: Seal the Secret into a SealedSecret
      shell: |
        set -euo pipefail
        cat {{ demo_secret_manifest }} \
        | {{ kubeseal_bin }} --format=yaml --cert {{ controller_cert }} \
        > {{ demo_sealed_manifest }}
      args:
        executable: /bin/bash
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Apply SealedSecret (idempotent)
      command: "kubectl apply -f {{ demo_sealed_manifest }}"
      register: apply_sealed
      changed_when: "'created' in (apply_sealed.stdout|default('')) or 'configured' in (apply_sealed.stdout|default(''))"
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Wait for decrypted Secret to appear
      shell: |
        set -euo pipefail
        kubectl -n {{ demo_ns }} get secret {{ demo_secret_name }} -o name
      args:
        executable: /bin/bash
      register: have_secret
      retries: 30
      delay: 3
      until: have_secret.rc == 0
      changed_when: false
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"
# OPRAVA kubeseal + pokračovanie
- name: Fix kubeseal and seal demo secret
  hosts: k3s-master
  gather_facts: false
  vars:
    kubeconfig_path: /etc/rancher/k3s/k3s.yaml
    kubeseal_version: v0.24.5
    kubeseal_url: "https://github.com/bitnami-labs/sealed-secrets/releases/download/{{ kubeseal_version }}/kubeseal-{{ kubeseal_version }}-linux-amd64.tar.gz"
    kubeseal_tar: /tmp/kubeseal.tar.gz
    kubeseal_bin: /usr/local/bin/kubeseal

    controller_namespace: kube-system
    controller_name: sealed-secrets
    controller_cert: /tmp/sealed-secrets.pem

    demo_ns: apps
    demo_secret_name: demo-secret
    demo_secret_manifest: "/tmp/{{ demo_secret_name }}.yaml"
    demo_sealed_manifest: "/tmp/{{ demo_secret_name }}-sealed.yaml"
    demo_username: demo
    demo_password: "S3cretP@ssw0rd!"

  tasks:
    - name: Check current kubeseal works
      command: "{{ kubeseal_bin }} --version"
      register: kubeseal_ver
      failed_when: false
      changed_when: false

    - name: Remove broken kubeseal (if needed)
      file:
        path: "{{ kubeseal_bin }}"
        state: absent
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Download kubeseal tarball (fresh)
      get_url:
        url: "{{ kubeseal_url }}"
        dest: "{{ kubeseal_tar }}"
        mode: '0644'
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Extract kubeseal
      unarchive:
        src: "{{ kubeseal_tar }}"
        dest: /tmp
        remote_src: true
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Install kubeseal to /usr/local/bin
      copy:
        src: /tmp/kubeseal
        dest: "{{ kubeseal_bin }}"
        mode: '0755'
        remote_src: true
      when: kubeseal_ver.rc != 0 or ('kubeseal' not in (kubeseal_ver.stdout|default('')))

    - name: Verify kubeseal installed
      command: "{{ kubeseal_bin }} --version"
      changed_when: false

    - name: Fetch controller public cert (PEM)
      shell: |
        set -euo pipefail
        {{ kubeseal_bin }} \
          --controller-name {{ controller_name }} \
          --controller-namespace {{ controller_namespace }} \
          --fetch-cert > {{ controller_cert }}
      args:
        executable: /bin/bash
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Ensure namespace exists for demo
      command: "kubectl get ns {{ demo_ns }}"
      register: ns_check
      failed_when: false
      changed_when: false
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Create namespace if missing
      command: "kubectl create ns {{ demo_ns }}"
      when: ns_check.rc != 0
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Render demo Secret manifest (unencrypted, local temp)
      copy:
        dest: "{{ demo_secret_manifest }}"
        mode: '0600'
        content: |
          apiVersion: v1
          kind: Secret
          metadata:
            name: {{ demo_secret_name }}
            namespace: {{ demo_ns }}
          type: Opaque
          stringData:
            username: "{{ demo_username }}"
            password: "{{ demo_password }}"

    - name: Seal the Secret into a SealedSecret
      shell: |
        set -euo pipefail
        cat {{ demo_secret_manifest }} \
        | {{ kubeseal_bin }} --format=yaml --cert {{ controller_cert }} \
        > {{ demo_sealed_manifest }}
      args:
        executable: /bin/bash
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Apply SealedSecret (idempotent)
      command: "kubectl apply -f {{ demo_sealed_manifest }}"
      register: apply_sealed
      changed_when: "'created' in (apply_sealed.stdout|default('')) or 'configured' in (apply_sealed.stdout|default(''))"
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Wait for decrypted Secret to appear
      shell: |
        set -euo pipefail
        kubectl -n {{ demo_ns }} get secret {{ demo_secret_name }} -o name
      args:
        executable: /bin/bash
      register: have_secret
      retries: 30
      delay: 3
      until: have_secret.rc == 0
      changed_when: false
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"
		
Ako otestujem teraz spravnost instalacie seal secret ?

Verification — end-to-end test

We validated three things:

  1. Controller & CRD are ready:
kubectl -n kube-system rollout status deploy/sealed-secrets --timeout=180s
kubectl get crd sealedsecrets.bitnami.com
kubeseal --version
  1. The decrypted Secret exists and is managed by the controller:
kubectl -n apps get secret demo-secret -o name
kubectl -n apps get secret demo-secret \
  -o jsonpath='{.metadata.annotations.sealedsecrets\.bitnami\.com/managed}'; echo
# expected: true
  1. Recreation after deletion (controller re-materializes the Secret):
kubectl -n apps get secret demo-secret -o jsonpath='{.metadata.resourceVersion}'
kubectl -n apps delete secret demo-secret
# wait & confirm it reappears
for i in {1..20}; do
  kubectl -n apps get secret demo-secret >/dev/null 2>&1 && echo "OK (recreated)" && break
  sleep 3
done

(Optional) Compact verification as an Ansible playbook:

---
- name: Day 12 | Verify Sealed Secrets installation & round-trip
  hosts: k3s-master
  gather_facts: false
  vars:
    kubeconfig_path: /etc/rancher/k3s/k3s.yaml
    controller_namespace: kube-system
    controller_name: sealed-secrets
    demo_ns: apps
    demo_secret_name: demo-secret
  tasks:
    - name: Check CRD exists
      command: kubectl get crd sealedsecrets.bitnami.com
      changed_when: false
      environment: { KUBECONFIG: "{{ kubeconfig_path }}" }

    - name: Wait for controller rollout
      command: kubectl -n {{ controller_namespace }} rollout status deploy/{{ controller_name }} --timeout=120s
      changed_when: false
      environment: { KUBECONFIG: "{{ kubeconfig_path }}" }

    - name: Check kubeseal works
      command: kubeseal --version
      register: kubeseal_ver
      changed_when: false
      failed_when: kubeseal_ver.rc != 0

    - name: Verify SealedSecret exists
      command: kubectl -n {{ demo_ns }} get sealedsecret {{ demo_secret_name }} -o name
      changed_when: false
      environment: { KUBECONFIG: "{{ kubeconfig_path }}" }

    - name: Verify decrypted Secret exists
      command: kubectl -n {{ demo_ns }} get secret {{ demo_secret_name }} -o name
      changed_when: false
      environment: { KUBECONFIG: "{{ kubeconfig_path }}" }

    - name: Check managed annotation on Secret
      command: >
        kubectl -n {{ demo_ns }} get secret {{ demo_secret_name }}
        -o jsonpath={{'{'}}.metadata.annotations.sealedsecrets\.bitnami\.com/managed{{'}'}}
      register: managed_anno
      changed_when: false
      environment: { KUBECONFIG: "{{ kubeconfig_path }}" }

    - name: Assert managed annotation present and true
      assert:
        that:
          - managed_anno.stdout | lower == "true"
        fail_msg: "Secret is not marked as managed by sealed-secrets controller."
        success_msg: "Secret is managed by sealed-secrets controller."

    - name: Capture current Secret resourceVersion
      command: kubectl -n {{ demo_ns }} get secret {{ demo_secret_name }} -o jsonpath='{.metadata.resourceVersion}'
      register: secret_rv_before
      changed_when: false
      environment: { KUBECONFIG: "{{ kubeconfig_path }}" }

    - name: Delete Secret (controller should recreate it)
      command: kubectl -n {{ demo_ns }} delete secret {{ demo_secret_name }}
      environment: { KUBECONFIG: "{{ kubeconfig_path }}" }

    - name: Wait for Secret to reappear
      command: kubectl -n {{ demo_ns }} get secret {{ demo_secret_name }} -o name
      register: have_secret
      retries: 20
      delay: 3
      until: have_secret.rc == 0
      changed_when: false
      environment: { KUBECONFIG: "{{ kubeconfig_path }}" }

    - name: Compare resourceVersion (should differ after recreation)
      command: kubectl -n {{ demo_ns }} get secret {{ demo_secret_name }} -o jsonpath='{.metadata.resourceVersion}'
      register: secret_rv_after
      changed_when: false
      environment: { KUBECONFIG: "{{ kubeconfig_path }}" }

    - name: Assert resourceVersion changed
      assert:
        that:
          - secret_rv_before.stdout != secret_rv_after.stdout
        fail_msg: "Secret did not get recreated/updated after deletion."
        success_msg: "Secret was recreated by the controller (resourceVersion changed)."

Key takeaways

  • Pinning a known-good kubeseal release avoids tar/packaging mismatches.
  • The controller public certificate is required to seal manifests offline.
  • The name + namespace pair is part of the encryption scope—changing either breaks decryption.
  • The controller will recreate Secrets from SealedSecrets—great for GitOps workflows.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.