HomeLab – Seal TLS Secrets with Bitnami Sealed Secrets + SSH Key for GitOps Commits – Day 13

Goal: Stop creating TLS secrets by hand and keep them encrypted in Git. Also enable k3s-master to push changes back to GitHub over SSH using our existing deploy key.

What we built

• Sealed an existing TLS secret (apps/nginx-tls) with the Sealed Secrets controller’s public certificate.
• Committed the resulting SealedSecret manifest to our GitOps repo so Argo CD can manage it declaratively.
• Installed our existing GitHub deploy SSH key onto k3s-master and validated SSH/push connectivity.

---

Playbook 1 – Seal an existing TLS Secret & commit to Git

File: ansible/day13_seal_tls_secret.yml

---
# Seal an existing TLS Secret and commit to Git (runs on k3s-master)

- name: Seal TLS secret and commit to Git
  hosts: k3s-master
  gather_facts: false

  vars:
    # --- Cluster access ---
    kubeconfig_path: /etc/rancher/k3s/k3s.yaml

    # --- Sealed Secrets controller info (Bitnami) ---
    controller_namespace: kube-system
    controller_name: sealed-secrets
    kubeseal_bin: /usr/local/bin/kubeseal
    controller_cert: /tmp/sealed-secrets.pem

    # --- Which secret are we sealing? ---
    source_secret_ns: apps
    source_secret_name: nginx-tls

    # --- Where to write the sealed manifest inside the Git repo on k3s-master ---
    git_repo_dir: "/home/stackadmin/fullstackhomelab"
    sealed_output_relpath: "gitops/secrets/apps/nginx/nginx-tls-sealed.yaml"
    sealed_output_abspath: "{{ git_repo_dir }}/{{ sealed_output_relpath }}"

    # --- Commit/push settings ---
    git_user_name: "Stackadmin CI"
    git_user_email: "ci@fullstackhomelab.local"
    git_branch: "main"
    git_remote: "origin"

    # If the SSH key is already installed on k3s-master:
    ssh_key_path: "/home/stackadmin/.ssh/argocd_deploy"
    git_ssh_command: "ssh -i {{ ssh_key_path }} -o StrictHostKeyChecking=yes"

  tasks:
    - name: Ensure kubeseal is installed
      command: "{{ kubeseal_bin }} --version"
      register: kubeseal_ver
      changed_when: false

    - name: Fetch Sealed Secrets controller public cert (PEM)
      shell: |
        set -euo pipefail
        {{ kubeseal_bin }} \
          --controller-name {{ controller_name }} \
          --controller-namespace {{ controller_namespace }} \
          --fetch-cert > {{ controller_cert }}
      args:
        executable: /bin/bash
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Ensure target directory inside repo exists
      file:
        path: "{{ (sealed_output_abspath | dirname) }}"
        state: directory
        mode: "0755"

    - name: Seal the existing TLS Secret directly from the cluster
      shell: |
        set -euo pipefail
        kubectl -n {{ source_secret_ns }} get secret {{ source_secret_name }} -o yaml \
        | {{ kubeseal_bin }} --format=yaml --cert {{ controller_cert }} \
        > {{ sealed_output_abspath }}
      args:
        executable: /bin/bash
      environment:
        KUBECONFIG: "{{ kubeconfig_path }}"

    - name: Configure git identity (idempotent)
      shell: |
        set -e
        git config user.name "{{ git_user_name }}"
        git config user.email "{{ git_user_email }}"
      args:
        chdir: "{{ git_repo_dir }}"
        executable: /bin/bash
      changed_when: false

    - name: git add the sealed secret
      shell: |
        set -e
        git add "{{ sealed_output_relpath }}"
      args:
        chdir: "{{ git_repo_dir }}"
        executable: /bin/bash

    - name: git commit (no-op if nothing changed)
      shell: |
        set -e
        git commit -m "Day 13: seal {{ source_secret_ns }}/{{ source_secret_name }} as SealedSecret" || true
      args:
        chdir: "{{ git_repo_dir }}"
        executable: /bin/bash
      register: commit_out
      changed_when: "' files changed' in commit_out.stdout or ' create mode ' in commit_out.stdout"

    - name: git push (requires GitHub Deploy Key with WRITE)
      shell: |
        set -e
        GIT_SSH_COMMAND='{{ git_ssh_command }}' \
          git push {{ git_remote }} {{ git_branch }}
      args:
        chdir: "{{ git_repo_dir }}"
        executable: /bin/bash
      register: push_out
      changed_when: false

    - name: Show commit/push results
      debug:
        msg:
          - "commit: {{ commit_out.stdout | default('') }}"
          - "push: {{ push_out.stdout | default('') }}"

What it does

• Pulls the live apps/nginx-tls secret from the cluster.
• Seals it with kubeseal using the controller’s public cert.
• Writes gitops/secrets/apps/nginx/nginx-tls-sealed.yaml inside the repo, commits it, and pushes to origin main over SSH.

Troubleshooting

• If push fails with Permission denied (publickey), confirm the SSH key on k3s-master matches your GitHub Deploy Key.
• If push is denied, change the GitHub Deploy Key from Read → Write.

---

Playbook 2 – Install existing GitHub Deploy Key onto k3s-master

File: ansible/day13_git_ssh_key_setup.yml

---
# Day 13 | Install existing GitHub SSH deploy key onto k3s-master (no key generation)

- name: Install existing Git SSH key and test connectivity
  hosts: k3s-master
  gather_facts: false

  vars:
    # === Paths on the CONTROL NODE (your Kubuntu 25) ===
    local_private_key_path: "/home/mocco/.ssh/argocd_deploy"       # existing private key on your laptop
    local_public_key_path:  "/home/mocco/.ssh/argocd_deploy.pub"   # existing public key on your laptop

    # === Destination on k3s-master ===
    remote_user: "stackadmin"
    remote_ssh_dir: "/home/{{ remote_user }}/.ssh"
    remote_key_path: "{{ remote_ssh_dir }}/argocd_deploy"
    remote_known_hosts: "{{ remote_ssh_dir }}/known_hosts"

    # === Optional local clone on k3s-master (if present) ===
    git_repo_url: "git@github.com:moccosvk/fullstackhomelab.git"
    git_repo_dir: "/home/{{ remote_user }}/fullstackhomelab"   # adjust if different

    do_git_dry_run_push: true
    git_user_name: "Stackadmin CI"
    git_user_email: "ci@fullstackhomelab.local"

  tasks:
    - name: Verify local keys exist (on control node)
      delegate_to: localhost
      run_once: true
      stat:
        path: "{{ item }}"
      register: key_stats
      loop:
        - "{{ local_private_key_path }}"
        - "{{ local_public_key_path }}"

    - name: Fail if local private/public key is missing
      when: >
        (key_stats.results[0].stat.exists is not defined or not key_stats.results[0].stat.exists)
        or
        (key_stats.results[1].stat.exists is not defined or not key_stats.results[1].stat.exists)
      fail:
        msg: >
          Missing local key files. Check local_private_key_path and local_public_key_path:
          {{ local_private_key_path }}, {{ local_public_key_path }}

    - name: Create ~/.ssh on k3s-master
      become: true
      file:
        path: "{{ remote_ssh_dir }}"
        state: directory
        owner: "{{ remote_user }}"
        group: "{{ remote_user }}"
        mode: "0700"

    - name: Copy private key to k3s-master
      become: true
      copy:
        src: "{{ local_private_key_path }}"
        dest: "{{ remote_key_path }}"
        owner: "{{ remote_user }}"
        group: "{{ remote_user }}"
        mode: "0600"

    - name: Copy public key to k3s-master (for reference)
      become: true
      copy:
        src: "{{ local_public_key_path }}"
        dest: "{{ remote_key_path }}.pub"
        owner: "{{ remote_user }}"
        group: "{{ remote_user }}"
        mode: "0644"

    - name: Add GitHub to known_hosts (rsa/ecdsa/ed25519)
      become: true
      shell: |
        set -euo pipefail
        touch "{{ remote_known_hosts }}"
        chown {{ remote_user }}:{{ remote_user }} "{{ remote_known_hosts }}"
        chmod 0644 "{{ remote_known_hosts }}"
        ssh-keyscan -t rsa,ecdsa,ed25519 github.com >> "{{ remote_known_hosts }}" 2>/dev/null || true
      args:
        executable: /bin/bash

    - name: Optionally set git identity if repo exists
      become: true
      shell: |
        set -e
        if [ -d "{{ git_repo_dir }}/.git" ]; then
          git config user.name "{{ git_user_name }}"
          git config user.email "{{ git_user_email }}"
        fi
      args:
        chdir: "{{ git_repo_dir }}"
        executable: /bin/bash
      failed_when: false
      changed_when: false

    - name: Optionally switch remote URL to SSH if repo exists
      become: true
      shell: |
        set -e
        if [ -d "{{ git_repo_dir }}/.git" ]; then
          git remote set-url origin "{{ git_repo_url }}"
        fi
      args:
        chdir: "{{ git_repo_dir }}"
        executable: /bin/bash
      failed_when: false
      changed_when: false

    - name: Test SSH connectivity to GitHub
      become: true
      shell: |
        set -e
        sudo -u {{ remote_user }} \
          bash -lc "GIT_SSH_COMMAND='ssh -i {{ remote_key_path }} -o StrictHostKeyChecking=yes' ssh -T git@github.com || true"
      args:
        executable: /bin/bash
      register: ssh_test
      changed_when: false

    - name: Show SSH test output
      debug:
        var: ssh_test.stdout

    - name: Optional dry-run push (verifies WRITE access for the key)
      when: do_git_dry_run_push | bool
      become: true
      shell: |
        set -e
        if [ -d .git ]; then
          sudo -u {{ remote_user }} \
            bash -lc "GIT_SSH_COMMAND='ssh -i {{ remote_key_path }} -o StrictHostKeyChecking=yes' git push --dry-run origin HEAD"
        fi
      args:
        chdir: "{{ git_repo_dir }}"
        executable: /bin/bash
      register: dry_run
      failed_when: false
      changed_when: false

    - name: Dry-run push output
      when: do_git_dry_run_push | bool
      debug:
        var: dry_run.stdout

What it does

• Copies your existing argocd_deploy keypair from the control node to k3s-master.
• Bootstraps known_hosts with GitHub host keys and validates SSH connectivity.
• Optionally performs a git push --dry-run to verify WRITE permissions.

Important: We switched the GitHub Deploy Key from Read → Write so the push could succeed.

---

How to test it end-to-end

1. Run Playbook 2 first (SSH key install):
   ansible-playbook -i ansible/inventory/hosts.ini ansible/day13_git_ssh_key_setup.yml
2. Run Playbook 1 (seal & commit):
   ansible-playbook -i ansible/inventory/hosts.ini ansible/day12_seal_tls_secret.yml
3. Verify the sealed manifest appeared in Git (gitops/secrets/apps/nginx/nginx-tls-sealed.yaml) and that Argo CD applies a decrypted Secret in apps namespace.

Why this matters

With Sealed Secrets and SSH push from the cluster, your TLS materials live encrypted at rest in Git, and all changes flow through GitOps. It’s repeatable, auditable, and production-friendly.