HomeLab – Argo CD GitOps: SSH Repo Secret, Connectivity Self-Test, Application Apply & Health Fixes – Day 11

DevOps, Homelab For Devops, Linux

In Day 11 we wired Argo CD to a private GitHub repository via SSH, validated connectivity from the argocd-repo-server pod, applied a Bitnami NGINX Helm chart as an Argo CD Application, and resolved a degraded health state by removing legacy resources and stabilizing TLS/ingress. Below are the exact Ansible playbooks and troubleshooting steps we used, end-to-end.

Prerequisites

  • Functional K3s cluster with Traefik (ingress class traefik).
  • Namespace argocd with Argo CD installed and accessible.
  • Namespace apps exists (or allow Argo CD to create it via CreateNamespace=true).
  • Valid TLS secret nginx-tls in apps (we used mkcert in earlier days).
  • GitHub repo (SSH URL) with at least one commit (HEAD must exist).

Playbook 1 – Create/Update Argo CD Repository Secret (SSH)

File: ansible/day11_argocd_repo_ssh.yml

---
# Day 11 | Create/Update Argo CD repo secret with SSH deploy key
# Run: ansible-playbook -i ansible/inventory/hosts.ini ansible/day11_argocd_repo_ssh.yml

- name: Day 11 | Argo CD repo (SSH) secret
  hosts: k3s-master
  gather_facts: false
  vars:
    # === EDIT ME ===
    repo_url: "git@github.com:moccosvk/fullstackhomelab.git"
    repo_name: "fullstackhomelab"
    secret_name: "repo-fullstackhomelab"
    namespace: "argocd"

    # Private deploy key on the CONTROL host (not on k3s-master)
    # Ensure mode 600 and it is readable.
    ssh_private_key_path: "~/.ssh/argocd_deploy"

    # Allow during bootstrap; later replace with 'known_hosts'
    insecure_ignore_hostkey: "true"

    remote_manifest: "/tmp/argocd-repo-secret.yaml"

  tasks:
    - name: "Read SSH private key from control host"
      ansible.builtin.slurp:
        src: "{{ ssh_private_key_path | expanduser }}"
      register: key_b64
      delegate_to: localhost

    - name: "Fail if key unreadable"
      ansible.builtin.fail:
        msg: "Cannot read SSH key at {{ ssh_private_key_path }} on control host."
      when: key_b64 is not defined or key_b64.content is not defined

    - name: "Build Secret manifest (stringData with raw multi-line key)"
      ansible.builtin.copy:
        dest: "{{ remote_manifest }}"
        mode: "0644"
        content: |
          apiVersion: v1
          kind: Secret
          metadata:
            name: {{ secret_name }}
            namespace: {{ namespace }}
            labels:
              argocd.argoproj.io/secret-type: repository
          type: Opaque
          stringData:
            name: "{{ repo_name }}"
            type: "git"
            url: "{{ repo_url }}"
            insecureIgnoreHostKey: "{{ insecure_ignore_hostkey }}"
            sshPrivateKey: |
{{ (key_b64.content | b64decode) | indent(14) }}

    - name: "kubectl apply repo secret"
      ansible.builtin.command:
        cmd: kubectl -n {{ namespace }} apply -f {{ remote_manifest }}
      register: apply_out
      changed_when: "'created' in apply_out.stdout or 'configured' in apply_out.stdout"

    - name: "Show result"
      ansible.builtin.debug:
        var: apply_out.stdout

Playbook 2 – Self-Test Repo Connectivity from Repo-Server Pod

File: ansible/day11_argocd_repo_selftest.yml

---
# Day 11 | Self-test repository connectivity from argocd-repo-server Pod
# Run: ansible-playbook -i ansible/inventory/hosts.ini ansible/day11_argocd_repo_selftest.yml

- name: Day 11 | Repo SSH self-test
  hosts: k3s-master
  gather_facts: false
  vars:
    namespace: "argocd"
    secret_name: "repo-fullstackhomelab"
    repo_url: "git@github.com:moccosvk/fullstackhomelab.git"

  tasks:
    - name: "Get repo-server Pod name"
      ansible.builtin.command:
        cmd: kubectl -n {{ namespace }} get pod -l app.kubernetes.io/name=argocd-repo-server -o jsonpath='{.items[0].metadata.name}'
      register: pod
      changed_when: false

    - name: "Extract private key from Secret"
      ansible.builtin.command:
        cmd: kubectl -n {{ namespace }} get secret {{ secret_name }} -o jsonpath='{.data.sshPrivateKey}'
      register: keydata
      changed_when: false

    - name: "Run git ls-remote using the secret’s key"
      ansible.builtin.shell: |
        set -euo pipefail
        KEY="$(printf '%s' '{{ keydata.stdout }}' | base64 -d)"
        kubectl -n {{ namespace }} exec -i "{{ pod.stdout }}" -- sh -lc '
          apk add --no-cache git openssh-client >/dev/null 2>&1 || true
          cat > /tmp/k <

Playbook 3 – Apply Argo CD Application (Bitnami NGINX)

File: ansible/day11_argocd_app_apply.yml

---
# Day 11 | Apply Argo CD Application for NGINX (GitOps)
# Run: ansible-playbook -i ansible/inventory/hosts.ini ansible/day11_argocd_app_apply.yml

- name: Day 11 | Apply Argo CD Application for NGINX (GitOps)
  hosts: k3s-master
  gather_facts: false
  vars:
    argocd_ns: "argocd"
    dest_ns: "apps"
    app_name: "apps-nginx"
    app_manifest: "/tmp/apps-nginx.yaml"

  tasks:
    - name: "Write Application manifest"
      ansible.builtin.copy:
        dest: "{{ app_manifest }}"
        mode: "0644"
        content: |
          apiVersion: argoproj.io/v1alpha1
          kind: Application
          metadata:
            name: {{ app_name }}
            namespace: {{ argocd_ns }}
          spec:
            project: default
            destination:
              namespace: {{ dest_ns }}
              server: https://kubernetes.default.svc
            source:
              repoURL: https://charts.bitnami.com/bitnami
              chart: nginx
              targetRevision: "21.*"
              helm:
                releaseName: web-gitops-nginx
                values: |
                  image:
                    tag: latest
                  replicaCount: 2
                  service:
                    type: ClusterIP
                  ingress:
                    enabled: true
                    ingressClassName: traefik
                    hostname: nginx.apps.lan
                    tls: true
                    extraTls:
                      - hosts: [nginx.apps.lan]
                        secretName: nginx-tls
                    annotations:
                      "traefik.ingress.kubernetes.io/router.middlewares": "apps-https-redirect@kubernetescrd"
            syncPolicy:
              automated:
                prune: true
                selfHeal: true
              syncOptions:
                - CreateNamespace=true

    - name: "Apply Application"
      ansible.builtin.command:
        cmd: kubectl -n {{ argocd_ns }} apply -f {{ app_manifest }}
      register: apply_out
      changed_when: "'created' in apply_out.stdout or 'configured' in apply_out.stdout"

    - name: "Show result"
      ansible.builtin.debug:
        var: apply_out.stdout

    - name: "Wait for Synced (up to 5m)"
      ansible.builtin.command:
        cmd: kubectl -n {{ argocd_ns }} wait application/{{ app_name }} --for=condition=Synced --timeout=300s
      register: sync_wait
      changed_when: false

    - name: "Wait for Healthy (poll every 10s, up to 10m)"
      ansible.builtin.command:
        cmd: kubectl -n {{ argocd_ns }} get application {{ app_name }} -o jsonpath={.status.health.status}
      register: health
      until: health.stdout == "Healthy"
      retries: 60
      delay: 10
      changed_when: false

    - name: "Show final status"
      ansible.builtin.debug:
        msg: "App status => Sync={{ (lookup('ansible.builtin.pipe','kubectl -n ' + argocd_ns + ' get application ' + app_name + ' -o jsonpath={.status.sync.status}')) }}, Health={{ health.stdout }}"

Playbook 4 – Fix Degraded App (Remove Legacy & Re-Wait)

File: ansible/day11_argocd_app_fix.yml

---
# Day 11 | Fix & wait Argo CD Application (NGINX)
# Run: ansible-playbook -i ansible/inventory/hosts.ini ansible/day11_argocd_app_fix.yml

- name: Day 11 | Fix Argo CD app (remove legacy and wait healthy)
  hosts: k3s-master
  gather_facts: false
  vars:
    argocd_ns: "argocd"
    dest_ns: "apps"
    app_name: "apps-nginx"
    tls_secret: "nginx-tls"

  tasks:
    - name: "Verify TLS secret exists in apps ({{ tls_secret }})"
      ansible.builtin.command:
        cmd: kubectl -n {{ dest_ns }} get secret {{ tls_secret }} -o name
      register: tlscheck
      failed_when: tlscheck.rc != 0
      changed_when: false

    - name: "Delete legacy non-GitOps resources (ignore if absent)"
      ansible.builtin.shell: |
        set -e
        kubectl -n {{ dest_ns }} delete deploy/web-nginx svc/web-nginx ingress/web-nginx --ignore-not-found
        kubectl -n {{ dest_ns }} delete deploy/web-nginx-gitops svc/web-nginx-gitops ingress/web-nginx-gitops --ignore-not-found
      register: del_out
      changed_when: false

    - name: "Re-apply Application (idempotent)"
      ansible.builtin.command:
        cmd: kubectl -n {{ argocd_ns }} apply -f /tmp/apps-nginx.yaml
      register: apply_out
      changed_when: "'created' in apply_out.stdout or 'configured' in apply_out.stdout"

    - name: "Show apply result"
      ansible.builtin.debug:
        var: apply_out.stdout

    - name: "Wait for Synced"
      ansible.builtin.command:
        cmd: kubectl -n {{ argocd_ns }} wait application/{{ app_name }} --for=condition=Synced --timeout=300s
      changed_when: false

    - name: "Wait for Healthy (retry)"
      ansible.builtin.command:
        cmd: kubectl -n {{ argocd_ns }} get application {{ app_name }} -o jsonpath={.status.health.status}
      register: health
      until: health.stdout == "Healthy"
      retries: 60
      delay: 10
      changed_when: false

    - name: "Show final status"
      ansible.builtin.debug:
        msg: "App is {{ health.stdout }}"

(Optional) Playbook 5 – Replace insecureIgnoreHostKey with Pinned known_hosts

File: ansible/day11_argocd_known_hosts_replace.yml

---
# Day 11 | Replace insecureIgnoreHostKey with pinned known_hosts
# Run: ansible-playbook -i ansible/inventory/hosts.ini ansible/day11_argocd_known_hosts_replace.yml

- name: Day 11 | Secure repo secret with known_hosts
  hosts: k3s-master
  gather_facts: false
  vars:
    namespace: "argocd"
    secret_name: "repo-fullstackhomelab"
    remote_manifest: "/tmp/argocd-repo-secret-secure.yaml"

  tasks:
    - name: "Fetch GitHub public host keys"
      ansible.builtin.shell: "ssh-keyscan -t rsa,ecdsa,ed25519 github.com 2>/dev/null"
      register: kh
      changed_when: false
      delegate_to: localhost

    - name: "Get current Secret (for url/name/type/sshPrivateKey)"
      ansible.builtin.command:
        cmd: kubectl -n {{ namespace }} get secret {{ secret_name }} -o json
      register: secjson
      changed_when: false

    - name: "Extract fields"
      ansible.builtin.set_fact:
        f_url: "{{ (secjson.stdout | from_json).data.url | b64decode }}"
        f_name: "{{ (secjson.stdout | from_json).data.name | b64decode }}"
        f_type: "{{ (secjson.stdout | from_json).data.type | b64decode }}"
        f_key_b64: "{{ (secjson.stdout | from_json).data.sshPrivateKey }}"

    - name: "Write secure Secret manifest (with known_hosts, without insecureIgnoreHostKey)"
      ansible.builtin.copy:
        dest: "{{ remote_manifest }}"
        mode: "0644"
        content: |
          apiVersion: v1
          kind: Secret
          metadata:
            name: {{ secret_name }}
            namespace: {{ namespace }}
            labels:
              argocd.argoproj.io/secret-type: repository
          type: Opaque
          data:
            name: "{{ f_name | b64encode }}"
            type: "{{ f_type | b64encode }}"
            url:  "{{ f_url  | b64encode }}"
            sshPrivateKey: "{{ f_key_b64 }}"
            known_hosts: "{{ kh.stdout | b64encode }}"

    - name: "Recreate Secret to drop old keys (clean replace)"
      ansible.builtin.shell: |
        set -e
        kubectl -n {{ namespace }} delete secret {{ secret_name }} --ignore-not-found
        kubectl -n {{ namespace }} apply -f {{ remote_manifest }}
      register: rep
      changed_when: "'created' in rep.stdout or 'configured' in rep.stdout or 'deleted' in rep.stdout"

    - name: "Show result"
      ansible.builtin.debug:
        var: rep.stdout

Troubleshooting We Hit (and How We Fixed It)

  1. YAML quoting errors (e.g., stray quotes in repo_url or Jinja braces inside shell args).
    Fix: Keep YAML keys simple; for long shell lines, move to copy: file content or use block scalars (|).
  2. “illegal base64 data at input byte 0” when creating Secrets.
    Fix: Use stringData for raw multi-line sshPrivateKey. Only use data for valid base64.
  3. Repo connectivity shows “remote repository is empty”.
    Fix: Push at least one commit to the repo so HEAD exists.
  4. Argo Application Degraded while old (non-GitOps) resources exist.
    Fix: Delete legacy web-nginx/web-nginx-gitops (deploy/svc/ingress) and let Argo own fresh resources.
  5. ImagePullBackOff with pinned tag 1.29.1-debian-12-r0.
    Fix: Switch to image.tag: latest (or any existing tag).
  6. TLS/Ingress errors (nginx-tls missing or wrong SAN).
    Fix: Ensure nginx-tls exists in apps and the cert covers nginx.apps.lan. Restart Traefik if needed.
  7. SSH host key issues.
    Fix: Bootstrap with insecureIgnoreHostKey: "true", then replace with pinned known_hosts (Playbook 5).

Verification Commands (Quick Checks)

# App status
kubectl -n argocd get application apps-nginx -o wide

# Pods, Services, Ingress in apps
kubectl -n apps get deploy,rs,po,svc,ingress -l app.kubernetes.io/name=nginx -o wide

# Argo repo-server can reach Github via SSH:
kubectl -n argocd logs deploy/argocd-repo-server --tail=100

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.