HomeLab – Wiring Argo CD to GitOps (and fixing SSH + CRD hiccups) – Day 16

DevOps, Homelab For Devops, Linux

Goal: have Argo CD manage our GitOps paths end-to-end and make the pipeline fully idempotent via Ansible.

Today looked simple on paper (just “wire paths into Application(s)/ApplicationSet”), but we hit three practical issues:

  1. Argo CD reported app path does not exist for apps-root
  2. The repo server could not fetch our private GitHub repo over SSH (Permission denied (publickey))
  3. kubectl wait silently failed because I used the wrong resource name (singular vs plural CRD)

We fixed all three and finished with apps-root = Synced & Healthy.

Symptoms & clues

  • Application showed Unknown/Healthy with a comparison error:
Failed to load target state: failed to generate manifest...
gitops/argocd/apps: app path does not exist

Listing the repo proved the files exist:

git -C ~/fullstackhomelab ls-tree --name-only HEAD
gitops
test.txt

git -C ~/fullstackhomelab ls-tree -r --name-only HEAD | grep -E 'gitops|argocd|apps|root|application|kustomization'
gitops/argocd/apps/apps-root.yaml
gitops/argocd/apps/infra-tls.yaml
gitops/cert-manager/base/cluster-issuer.yaml

Argo CD controller logs confirmed the path mismatch and then later normalized spec updates:

Normalized app spec ... "gitops/argocd/apps: app path does not exist"

Git fetch failed from the node:

git -C ~/fullstackhomelab fetch origin
git@github.com: Permission denied (publickey)

The repo-server logs showed manifest cache hits (good) and that it could render other sources:

manifest cache hit ... RepoURL:git@github.com:moccosvk/fullstackhomelab.git, Path:gitops/infra/tls

kubectl wait kept timing out until we used the plural Group/Kind:

applications.argoproj.io/apps-root

Fix 1 — Correct the Application path and force refresh

I temporarily repointed apps-root and hard-refreshed:

kubectl -n argocd patch application apps-root --type=merge -p '{
  "spec": {
    "source": {
      "repoURL": "git@github.com:moccosvk/fullstackhomelab.git",
      "targetRevision": "main",
      "path": "gitops/argocd"
    },
    "syncPolicy": { "automated": { "prune": true, "selfHeal": true } }
  }
}'

kubectl -n argocd annotate application apps-root \
  argocd.argoproj.io/refresh=hard --overwrite

(Note: use the directory that actually contains your Kustomize/Helm entrypoint. For me, validating the tree with git ls-tree made it obvious.)

Fix 2 — Enable SSH access for GitHub from the cluster/node

We discovered the playbook didn’t push changes (no ssh-agent task, no key loaded), and manual fetch failed. Steps we validated:

# On the node:
ls -l ~/.ssh/argocd_deploy ~/.ssh/argocd_deploy.pub

# Optional: trust GitHub host key once (if not already)
ssh -o StrictHostKeyChecking=accept-new -T git@github.com || true

# Test the key works locally
ssh -i ~/.ssh/argocd_deploy -T git@github.com
# You should see "Hi <user>! You've successfully authenticated..."

Then ensure the repo’s remote uses SSH:

git -C ~/fullstackhomelab remote -v
git -C ~/fullstackhomelab remote set-url origin git@github.com:moccosvk/fullstackhomelab.git

Fix 3 — Use the plural CRD with kubectl wait

Argo Application is a CRD in argoproj.io/v1alpha1. The resource name for kubectl is plural:

kubectl api-resources | grep -i argoproj
# applicationsets, applications, appprojects ...

# Correct waits:
kubectl -n argocd wait applications.argoproj.io/apps-root \
  --for=jsonpath='{.status.sync.status}'=Synced --timeout=300s

kubectl -n argocd wait applications.argoproj.io/apps-root \
  --for=jsonpath='{.status.health.status}'=Healthy --timeout=300s

Result:

application.argoproj.io/apps-root condition met
application.argoproj.io/apps-root condition met

And the final status:

kubectl -n argocd get application apps-root -o wide
NAME        SYNC STATUS   HEALTH STATUS   REVISION                                   PROJECT
apps-root   Synced        Healthy         30d0585698a60663c5557c4ccec9bd31eae1e015   default

Useful kubectl and log commands from today

# Inspect the current sync/health/conditions quickly
kubectl -n argocd get application apps-root \
  -o jsonpath='{.status.sync.status}{"\n"}{.status.health.status}{"\n"}{.status.operationState.phase}{"\n"}{.status.conditions[*].message}{"\n"}'

# Get Argo CD pods
kubectl -n argocd get pods -l app.kubernetes.io/part-of=argocd

# Controller logs filtered by app
POD=$(kubectl -n argocd get pods \
  -l app.kubernetes.io/name=argocd-application-controller \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n argocd logs "$POD" --tail=200

# Repo-server logs
kubectl -n argocd logs deploy/argocd-repo-server --tail=200

# Watch Application status
kubectl -n argocd get application apps-root -w

The Ansible bits (final, idempotent tasks)

1) Make sure /bin/bash is used when relying on set -euo pipefail

We saw /bin/sh: set: Illegal option -o pipefail. Always set executable: /bin/bash for such tasks.

2) Idempotent origin URL (SSH)

# tasks/git_origin.yml
- name: Ensure origin remote points to SSH URL (idempotent)
  shell: |
    set -euo pipefail
    if git remote | grep -q '^origin$'; then
      git remote set-url origin "git@github.com:moccosvk/fullstackhomelab.git"
    else
      git remote add origin "git@github.com:moccosvk/fullstackhomelab.git"
    fi
  args:
    chdir: /home/stackadmin/fullstackhomelab
    executable: /bin/bash
  register: set_origin
  changed_when: set_origin.rc == 0

3) Push with ssh-agent (requires collection community.general)

Install once:
ansible-galaxy collection install community.general

# tasks/git_push.yml
- name: Push to origin main with ssh-agent (creates upstream if missing)
  community.general.ssh_agent:
    ssh_args: "-o StrictHostKeyChecking=accept-new"
    ssh_identity_add: yes
    ssh_identity_files:
      - /home/stackadmin/.ssh/argocd_deploy
  register: agent_env

- name: Ensure GitHub host key is known (first connect)
  shell: ssh -o StrictHostKeyChecking=accept-new -T git@github.com || true
  environment: "{{ agent_env.ssh_auth_sock | default({}) }}"
  changed_when: false

- name: Set upstream (if missing) and push
  shell: |
    set -euo pipefail
    git checkout -B main
    # If upstream is missing, --set-upstream will create it; otherwise it's a no-op
    git push --set-upstream origin main
  args:
    chdir: /home/stackadmin/fullstackhomelab
    executable: /bin/bash
  environment:
    SSH_AUTH_SOCK: "{{ agent_env.ssh_auth_sock }}"

(If you prefer zero external collections, you can wrap everything in ssh-agent bash -lc 'ssh-add ...; git push ...'. I kept the clean variant.)

4) Force Argo refresh (optional) and wait until Synced & Healthy

# tasks/argocd_wait.yml
- name: Hard refresh apps-root (optional)
  command: >
    kubectl -n argocd annotate application apps-root
    argocd.argoproj.io/refresh=hard --overwrite
  changed_when: false

# Option A: kubectl wait using plural CRD
- name: Wait for apps-root to be Synced
  command: >
    kubectl -n argocd wait applications.argoproj.io/apps-root
    --for=jsonpath='{.status.sync.status}'=Synced --timeout=300s
  changed_when: false

- name: Wait for apps-root to be Healthy
  command: >
    kubectl -n argocd wait applications.argoproj.io/apps-root
    --for=jsonpath='{.status.health.status}'=Healthy --timeout=300s
  changed_when: false

# Option B: Polling (alternative)
# - name: Poll apps-root until Synced & Healthy
#   shell: |
#     kubectl -n argocd get applications.argoproj.io apps-root \
#       -o jsonpath='{.status.sync.status}{" "}{.status.health.status}'
#   register: app_status
#   until: "'Synced Healthy' in app_status.stdout"
#   retries: 100
#   delay: 3
#   changed_when: false

5) Full Day 16 playbook (consolidated)

# ansible/day16_argocd_wire_gitops.yml
---
- name: Day 16 — Wire Argo CD to GitOps and push via SSH
  hosts: k3s-master
  become: true
  vars:
    repo_dir: /home/stackadmin/fullstackhomelab
    repo_url_ssh: git@github.com:moccosvk/fullstackhomelab.git
    deploy_key: /home/stackadmin/.ssh/argocd_deploy

  pre_tasks:
    - name: Ensure repo dir exists
      file:
        path: "{{ repo_dir }}"
        state: directory
        owner: stackadmin
        group: stackadmin
        mode: "0755"

  tasks:
    - name: Configure origin remote to SSH
      shell: |
        set -euo pipefail
        if git remote | grep -q '^origin$'; then
          git remote set-url origin "{{ repo_url_ssh }}"
        else
          git remote add origin "{{ repo_url_ssh }}"
        fi
      args:
        chdir: "{{ repo_dir }}"
        executable: /bin/bash
      become_user: stackadmin

    - name: Start ssh-agent and add deploy key
      community.general.ssh_agent:
        ssh_args: "-o StrictHostKeyChecking=accept-new"
        ssh_identity_add: yes
        ssh_identity_files:
          - "{{ deploy_key }}"
      register: agent_env
      become_user: stackadmin

    - name: Prime GitHub host key (first connect)
      shell: ssh -o StrictHostKeyChecking=accept-new -T git@github.com || true
      environment:
        SSH_AUTH_SOCK: "{{ agent_env.ssh_auth_sock }}"
      changed_when: false
      become_user: stackadmin

    - name: Push main (create upstream if missing)
      shell: |
        set -euo pipefail
        git checkout -B main
        git push --set-upstream origin main
      args:
        chdir: "{{ repo_dir }}"
        executable: /bin/bash
      environment:
        SSH_AUTH_SOCK: "{{ agent_env.ssh_auth_sock }}"
      become_user: stackadmin

    - name: Hard refresh apps-root (optional)
      command: >
        kubectl -n argocd annotate application apps-root
        argocd.argoproj.io/refresh=hard --overwrite
      changed_when: false

    - name: Wait for apps-root Synced
      command: >
        kubectl -n argocd wait applications.argoproj.io/apps-root
        --for=jsonpath='{.status.sync.status}'=Synced --timeout=300s
      changed_when: false

    - name: Wait for apps-root Healthy
      command: >
        kubectl -n argocd wait applications.argoproj.io/apps-root
        --for=jsonpath='{.status.health.status}'=Healthy --timeout=300s
      changed_when: false

Note: Install the ssh-agent collection before running the playbook:

ansible-galaxy collection install community.general

Sanity checks (end state)

kubectl -n argocd get application apps-root -o wide
# apps-root   Synced  Healthy  <commit-sha>  default

while true; do
  S=$(kubectl -n argocd get applications.argoproj.io apps-root -o jsonpath='{.status.sync.status}')
  H=$(kubectl -n argocd get applications.argoproj.io apps-root -o jsonpath='{.status.health.status}')
  [ "$S" = "Synced" ] && [ "$H" = "Healthy" ] && break
  sleep 3
done
echo "apps-root is Synced & Healthy"

Takeaways

  • When Argo says “path does not exist,” verify the repo tree and the Application spec.source.path actually point to a dir with a valid entrypoint (Kustomize/Helm/chart).
  • For private Git SSH, automate ssh-agent loading in Ansible and ensure host keys are accepted idempotently.
  • For Argo CD CRDs, kubectl wait must use the plural resource (applications.argoproj.io/<name>), not the singular.

Result: Argo CD now continuously reconciles from our Git repo, and our Day 16 playbook can push updates and block until the system is Synced & Healthy.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.