HomeLab – Making TLS Truly GitOps: Exporting ClusterIssuer, Certificate & Ingress to Git – Day 15

DevOps, Homelab For Devops, Linux

Today we closed the loop on TLS by exporting live Kubernetes resources (our ClusterIssuer, the Certificate, and the NGINX Ingress objects) into our Git repo so the entire configuration is declarative and versioned. We also hardened the Git workflow on the control host to reliably push via SSH.

What we accomplished

  • Cloned/updated the repo on the control node and ensured Git trusts the working directory (safe.directory fix).
  • Exported:
    • ClusterIssuer mkcert-cluster-issuer
    • Certificate nginx-tls (namespace apps)
    • Ingress objects pointing to nginx.apps.lan
  • Normalized YAML (removed managedFields and status) to keep diffs clean.
  • Committed and pushed changes using our already-provisioned deploy key.

Full Ansible playbook

File: ansible/day15_tls_export_to_git.yml

---
- name: Day 15 | Export TLS objects to Git (fully declarative)
  hosts: k3s-master
  gather_facts: false

  vars:
    kubeconfig_path: /etc/rancher/k3s/k3s.yaml

    # === Git repo settings (adjust if your paths differ) ===
    repo_owner_user: mocco
    repo_dir: /home/mocco/fullstackhomelab
    repo_remote_url: "git@github.com:moccosvk/fullstackhomelab.git"
    repo_branch: main

    # Use the deploy key we already set up in previous days (present on control host)
    ssh_key_path: /home/mocco/.ssh/argocd_deploy

    # Where to store exported manifests inside the repo
    out_dir_tls: "{{ repo_dir }}/gitops/tls"
    out_dir_apps: "{{ repo_dir }}/gitops/apps"

    # Resources to export
    clusterissuer_name: mkcert-cluster-issuer
    cert_namespace: apps
    cert_name: nginx-tls

    # Ingresses to export (namespace:name)
    ingress_list:
      - { ns: "apps", name: "web-gitops-nginx" }
      - { ns: "apps", name: "web-nginx-gitops" }

  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"

  tasks:
    - name: Ensure repo base directory exists
      file:
        path: "{{ repo_dir }}"
        state: directory
        owner: "{{ repo_owner_user }}"
        group: "{{ repo_owner_user }}"
        mode: "0755"

    - name: If repo is empty dir, git clone (idempotent)
      become_user: "{{ repo_owner_user }}"
      shell: |
        set -euo pipefail
        if [ ! -d .git ]; then
          git clone "{{ repo_remote_url }}" "{{ repo_dir }}"
        fi
      args:
        chdir: "{{ repo_dir }}"
      changed_when: false

    - name: Mark repo as safe.directory (avoid 'dubious ownership')
      become_user: "{{ repo_owner_user }}"
      command: "git config --global --add safe.directory {{ repo_dir }}"
      args:
        chdir: "{{ repo_dir }}"
      failed_when: false
      changed_when: false

    - name: Fix ownership recursively (just in case)
      file:
        path: "{{ repo_dir }}"
        state: directory
        owner: "{{ repo_owner_user }}"
        group: "{{ repo_owner_user }}"
        recurse: true

    - name: Ensure git remote URL is SSH (origin)
      become_user: "{{ repo_owner_user }}"
      shell: |
        set -euo pipefail
        if git remote get-url origin >/dev/null 2>&1; then
          git remote set-url origin "{{ repo_remote_url }}"
        else
          git remote add origin "{{ repo_remote_url }}"
        fi
      args:
        chdir: "{{ repo_dir }}"

    - name: Configure git identity (use replace-all to avoid multi-values)
      become_user: "{{ repo_owner_user }}"
      loop:
        - { key: "user.name",  value: "FullstackQuest Bot" }
        - { key: "user.email", value: "bot@fullstackquest.local" }
      command: "git config --replace-all {{ item.key }} '{{ item.value }}'"
      args:
        chdir: "{{ repo_dir }}"

    - name: Fetch & checkout branch (create if missing)
      become_user: "{{ repo_owner_user }}"
      shell: |
        set -euo pipefail
        git fetch origin || true
        if git show-ref --verify --quiet "refs/heads/{{ repo_branch }}"; then
          git checkout "{{ repo_branch }}"
          git pull --ff-only origin "{{ repo_branch }}" || true
        else
          git checkout -b "{{ repo_branch }}"
        fi
      args:
        chdir: "{{ repo_dir }}"

    - name: Create output directories
      file:
        path: "{{ item }}"
        state: directory
        owner: "{{ repo_owner_user }}"
        group: "{{ repo_owner_user }}"
        mode: "0755"
      loop:
        - "{{ out_dir_tls }}"
        - "{{ out_dir_apps }}"
        - "{{ out_dir_apps }}/ingress"

    # ---------- Export resources ----------
    - name: Export ClusterIssuer (raw)
      become_user: "{{ repo_owner_user }}"
      shell: |
        set -euo pipefail
        kubectl get clusterissuer "{{ clusterissuer_name }}" -o yaml > "{{ out_dir_tls }}/clusterissuer-{{ clusterissuer_name }}.raw.yaml"
      args:
        chdir: "{{ repo_dir }}"

    - name: Clean ClusterIssuer YAML (strip managedFields/status)
      become_user: "{{ repo_owner_user }}"
      shell: |
        set -euo pipefail
        awk '
          /managedFields:/ { skip=1 }
          skip==1 && NF==0 { next }
          skip==1 && $1!~/^ / { skip=0 }
          skip==1 { next }
          /^status:/ { s=1 }
          s==1 && NF==0 { s=0; next }
          s==1 { next }
          { print }
        ' "{{ out_dir_tls }}/clusterissuer-{{ clusterissuer_name }}.raw.yaml" > "{{ out_dir_tls }}/clusterissuer-{{ clusterissuer_name }}.yaml"
      args:
        chdir: "{{ repo_dir }}"

    - name: Remove raw file (ClusterIssuer)
      file:
        path: "{{ out_dir_tls }}/clusterissuer-{{ clusterissuer_name }}.raw.yaml"
        state: absent

    - name: Export Certificate (raw)
      become_user: "{{ repo_owner_user }}"
      shell: |
        set -euo pipefail
        kubectl -n "{{ cert_namespace }}" get certificate "{{ cert_name }}" -o yaml > "{{ out_dir_tls }}/certificate-{{ cert_namespace }}-{{ cert_name }}.raw.yaml"
      args:
        chdir: "{{ repo_dir }}"

    - name: Clean Certificate YAML (strip managedFields/status)
      become_user: "{{ repo_owner_user }}"
      shell: |
        set -euo pipefail
        awk '
          /managedFields:/ { skip=1 }
          skip==1 && NF==0 { next }
          skip==1 && $1!~/^ / { skip=0 }
          skip==1 { next }
          /^status:/ { s=1 }
          s==1 && NF==0 { s=0; next }
          s==1 { next }
          { print }
        ' "{{ out_dir_tls }}/certificate-{{ cert_namespace }}-{{ cert_name }}.raw.yaml" > "{{ out_dir_tls }}/certificate-{{ cert_namespace }}-{{ cert_name }}.yaml"
      args:
        chdir: "{{ repo_dir }}"

    - name: Remove raw file (Certificate)
      file:
        path: "{{ out_dir_tls }}/certificate-{{ cert_namespace }}-{{ cert_name }}.raw.yaml"
        state: absent

    - name: Export Ingresses (raw)
      become_user: "{{ repo_owner_user }}"
      loop: "{{ ingress_list }}"
      loop_control:
        loop_var: ing
      shell: |
        set -euo pipefail
        kubectl -n "{{ ing.ns }}" get ingress "{{ ing.name }}" -o yaml \
          > "{{ out_dir_apps }}/ingress/{{ ing.ns }}-{{ ing.name }}.raw.yaml"
      args:
        chdir: "{{ repo_dir }}"

    - name: Clean Ingress YAMLs (strip managedFields/status)
      become_user: "{{ repo_owner_user }}"
      loop: "{{ ingress_list }}"
      loop_control:
        loop_var: ing
      shell: |
        set -euo pipefail
        awk '
          /managedFields:/ { skip=1 }
          skip==1 && NF==0 { next }
          skip==1 && $1!~/^ / { skip=0 }
          skip==1 { next }
          /^status:/ { s=1 }
          s==1 && NF==0 { s=0; next }
          s==1 { next }
          { print }
        ' "{{ out_dir_apps }}/ingress/{{ ing.ns }}-{{ ing.name }}.raw.yaml" \
          > "{{ out_dir_apps }}/ingress/{{ ing.ns }}-{{ ing.name }}.yaml"
      args:
        chdir: "{{ repo_dir }}"

    - name: Remove raw files (Ingresses)
      file:
        path: "{{ out_dir_apps }}/ingress"
        state: directory
        recurse: true
      register: rm_ingress_dir

    - name: Recreate cleaned Ingress directory (ensure .yaml only)
      when: rm_ingress_dir is changed
      file:
        path: "{{ out_dir_apps }}/ingress"
        state: directory
        owner: "{{ repo_owner_user }}"
        group: "{{ repo_owner_user }}"
        mode: "0755"

    # ---------- Commit & push ----------
    - name: Git add changes
      become_user: "{{ repo_owner_user }}"
      command: "git add -A"
      args:
        chdir: "{{ repo_dir }}"

    - name: Commit (if there is anything to commit)
      become_user: "{{ repo_owner_user }}"
      shell: |
        set -euo pipefail
        if ! git diff --cached --quiet; then
          git commit -m "Day 15: export TLS to Git (ClusterIssuer, Certificate, Ingress)"
        fi
      args:
        chdir: "{{ repo_dir }}"

    - name: Push via SSH deploy key
      become_user: "{{ repo_owner_user }}"
      shell: |
        set -euo pipefail
        if [ -f "{{ ssh_key_path }}" ]; then
          GIT_SSH_COMMAND="ssh -i {{ ssh_key_path }} -o StrictHostKeyChecking=no" \
            git push origin "{{ repo_branch }}"
        else
          echo "WARN: SSH key {{ ssh_key_path }} not found; skipping push." >&2
        fi
      args:
        chdir: "{{ repo_dir }}"

Troubleshooting

“fatal: detected dubious ownership”

If you see this during git commands on the control host:

sudo chown -R stackadmin:stackadmin /home/stackadmin/fullstackhomelab
sudo -u stackadmin git config --global --add safe.directory /home/stackadmin/fullstackhomelab
# (optional, if you ever ran git as root)
sudo git config --global --add safe.directory /home/stackadmin/fullstackhomelab

Multiple user.name / user.email valuesUse --replace-all to avoid “cannot overwrite multiple values” errors:

git config --replace-all user.name  "FullstackQuest Bot"
git config --replace-all user.email "bot@fullstackquest.local"

Verify what we exported

tree gitops
# gitops/
# ├── tls
# │   ├── clusterissuer-mkcert-cluster-issuer.yaml
# │   └── certificate-apps-nginx-tls.yaml
# └── apps
#     └── ingress
#         ├── apps-web-gitops-nginx.yaml
#         └── apps-web-nginx-gitops.yaml

Why this matters

With cert-manager issuing TLS automatically (from Day 14) and Sealed Secrets covering sensitive data (Day 12), putting the source of truth for TLS (issuer, certificate objects, and ingress) in Git means:

  • Full reproducibility (any cluster can converge from Git).
  • Traceable changes (PRs + diffs on YAML).
  • No more snowflake resources created by hand.

Next time: we’ll wire these paths into our Application(s)/ApplicationSet so Argo CD manages them end-to-end.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.