HomeLab – GitOps with Argo CD on k3s – Day 10

Goal: Install Argo CD via Helm, expose it through Traefik with mkcert TLS, and bootstrap a first GitOps Application that auto-syncs a Bitnami nginx deployment. Control host: fullstacklab.site (user stackadmin); kubectl/helm run on k3s-master.

---

Step 1 — mkcert TLS for argocd.apps.lan and Kubernetes Secret

We mint a locally trusted certificate for argocd.apps.lan on the control host and rotate a TLS secret in the argocd namespace on the master.

---
# ansible/day10_mkcert_argocd.yml

# Play 1: mint mkcert certificate (localhost)
- name: Mint mkcert cert for argocd.apps.lan on control host
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    cert_dir: "{{ lookup('env','HOME') }}/.mkcert/argocd-apps-lan"
    cn: "argocd.apps.lan"
  tasks:
    - name: Ensure cert dir exists
      ansible.builtin.file:
        path: "{{ cert_dir }}"
        state: directory
        mode: "0755"
    - name: Install local Root CA (no-op if already installed)
      ansible.builtin.command: mkcert -install
      changed_when: false
    - name: Mint leaf cert for {{ cn }}
      ansible.builtin.command: >
        mkcert -cert-file {{ cert_dir }}/{{ cn }}.crt
               -key-file {{ cert_dir }}/{{ cn }}.key
               {{ cn }}
      args:
        creates: "{{ cert_dir }}/{{ cn }}.crt"

# Play 2: create namespace + TLS secret (k3s-master)
- name: Create argocd namespace and TLS secret from mkcert output
  hosts: k3s_master
  become: true
  vars:
    kubeconfig: "/etc/rancher/k3s/k3s.yaml"
    namespace: "argocd"
    secret_name: "argocd-tls"
    cn: "argocd.apps.lan"
    local_cert_dir: "{{ lookup('env','HOME') }}/.mkcert/argocd-apps-lan"
    remote_staging_dir: "/tmp/mkcert-argocd-apps-lan"
  environment:
    KUBECONFIG: "{{ kubeconfig }}"
  tasks:
    - name: Ensure namespace exists
      ansible.builtin.command: kubectl apply -f -
      args:
        stdin: |
          apiVersion: v1
          kind: Namespace
          metadata:
            name: {{ namespace }}

    - name: Create remote staging dir
      ansible.builtin.file:
        path: "{{ remote_staging_dir }}"
        state: directory
        mode: "0755"

    # robust transfer (works even if files are root-owned)
    - name: Read CRT from localhost (root)
      delegate_to: localhost
      become: true
      ansible.builtin.slurp:
        src: "{{ local_cert_dir }}/{{ cn }}.crt"
      register: crt_b64

    - name: Read KEY from localhost (root)
      delegate_to: localhost
      become: true
      ansible.builtin.slurp:
        src: "{{ local_cert_dir }}/{{ cn }}.key"
      register: key_b64

    - name: Write CRT on master
      ansible.builtin.copy:
        dest: "{{ remote_staging_dir }}/{{ cn }}.crt"
        mode: "0644"
        content: "{{ crt_b64.content | b64decode }}"

    - name: Write KEY on master
      ansible.builtin.copy:
        dest: "{{ remote_staging_dir }}/{{ cn }}.key"
        mode: "0600"
        content: "{{ key_b64.content | b64decode }}"

    - name: Create/Apply TLS secret (idempotent)
      ansible.builtin.shell: |
        kubectl -n {{ namespace }} create secret tls {{ secret_name }} \
          --cert={{ remote_staging_dir }}/{{ cn }}.crt \
          --key={{ remote_staging_dir }}/{{ cn }}.key \
          --dry-run=client -o yaml | kubectl apply -f -

Step 2 — Install Argo CD with Traefik Ingress + TLS

We deploy Argo CD via Helm, expose the UI at https://argocd.apps.lan/ through Traefik, and enable a simple HTTPS redirect middleware.

---
# ansible/day10_argocd_install.yml
- name: Install Argo CD via Helm with Traefik Ingress + TLS
  hosts: k3s_master
  become: true
  vars:
    kubeconfig: "/etc/rancher/k3s/k3s.yaml"
    namespace: "argocd"
    release_name: "argocd"
    ingress_host: "argocd.apps.lan"
    tls_secret: "argocd-tls"
    values_file: "/tmp/argocd-values.yaml"
  environment:
    KUBECONFIG: "{{ kubeconfig }}"
  tasks:
    - name: Ensure argo helm repo exists
      ansible.builtin.command: helm repo add argo https://argoproj.github.io/argo-helm
      register: add_repo
      failed_when: add_repo.rc not in [0,1]
      changed_when: add_repo.rc == 0

    - name: Helm repo update
      ansible.builtin.command: helm repo update

    - name: Render values.yaml for argo-cd
      ansible.builtin.copy:
        dest: "{{ values_file }}"
        mode: "0644"
        content: |
          crds:
            install: true
          server:
            service:
              type: ClusterIP
            ingress:
              enabled: true
              ingressClassName: traefik
              annotations:
                traefik.ingress.kubernetes.io/router.entrypoints: websecure
                traefik.ingress.kubernetes.io/router.tls: "true"
                traefik.ingress.kubernetes.io/router.middlewares: "argocd-https-redirect@kubernetescrd"
              hosts:
                - {{ ingress_host }}
              tls:
                - secretName: {{ tls_secret }}
                  hosts:
                    - {{ ingress_host }}
              paths:
                - /
              pathType: Prefix
            extraArgs:
              - --insecure

    - name: Helm upgrade --install argo-cd
      ansible.builtin.command: >
        helm upgrade --install {{ release_name }} argo/argo-cd -n {{ namespace }}
        -f {{ values_file }}

    - name: Create Traefik Middleware HTTPS redirect (idempotent)
      ansible.builtin.command: kubectl -n {{ namespace }} apply -f -
      args:
        stdin: |
          apiVersion: traefik.io/v1alpha1
          kind: Middleware
          metadata:
            name: https-redirect
          spec:
            redirectScheme:
              scheme: https
              permanent: true

    - name: Wait for Argo CD server to be ready
      ansible.builtin.command: kubectl -n {{ namespace }} rollout status deploy/{{ release_name }}-server --timeout=300s

    - name: Show Argo CD components
      ansible.builtin.command: kubectl -n {{ namespace }} get deploy,svc,ingress -o wide
      changed_when: false

Step 3 — Get the admin password

---
# ansible/day10_argocd_admin_pass.yml
- name: Get Argo CD initial admin password
  hosts: k3s_master
  become: true
  vars:
    kubeconfig: "/etc/rancher/k3s/k3s.yaml"
    namespace: "argocd"
  environment:
    KUBECONFIG: "{{ kubeconfig }}"
  tasks:
    - name: Read admin secret
      ansible.builtin.command: >
        kubectl -n {{ namespace }} get secret argocd-initial-admin-secret
        -o jsonpath="{.data.password}"
      register: pw_b64
      changed_when: false
    - name: Decode and print
      ansible.builtin.debug:
        msg: "Argo CD admin password: {{ pw_b64.stdout | b64decode }}"

Step 4 — Bootstrap a GitOps Application (Bitnami nginx)

We create an Argo CD Application resource that pulls the Bitnami nginx chart directly from GitHub and deploys it into apps. TLS/Ingress are aligned with our Traefik setup.

---
# ansible/day10_argocd_bootstrap_app.yml
- name: Bootstrap a GitOps Application (Bitnami nginx)
  hosts: k3s_master
  become: true
  vars:
    kubeconfig: "/etc/rancher/k3s/k3s.yaml"
    namespace: "argocd"
    app_name: "web-nginx-gitops"
    repo_url: "https://github.com/bitnami/charts"
    repo_path: "bitnami/nginx"
    dest_namespace: "apps"
    ingress_host: "nginx.apps.lan"
    tls_secret: "nginx-tls"
  environment:
    KUBECONFIG: "{{ kubeconfig }}"
  tasks:
    - name: Apply Argo CD Application
      ansible.builtin.command: kubectl -n {{ namespace }} apply -f -
      args:
        stdin: |
          apiVersion: argoproj.io/v1alpha1
          kind: Application
          metadata:
            name: {{ app_name }}
            namespace: {{ namespace }}
            finalizers:
              - resources-finalizer.argocd.argoproj.io
          spec:
            project: default
            source:
              repoURL: {{ repo_url }}
              path: {{ repo_path }}
              targetRevision: HEAD
              helm:
                values: |
                  image:
                    tag: latest
                  replicaCount: 2
                  service:
                    type: ClusterIP
                  ingress:
                    enabled: true
                    ingressClassName: traefik
                    hostname: {{ ingress_host }}
                    tls: true
                    extraTls:
                      - hosts: [{{ ingress_host }}]
                        secretName: {{ tls_secret }}
                    annotations:
                      "traefik.ingress.kubernetes.io/router.middlewares": "apps-https-redirect@kubernetescrd"
            destination:
              server: https://kubernetes.default.svc
              namespace: {{ dest_namespace }}
            syncPolicy:
              automated:
                prune: true
                selfHeal: true
              syncOptions:
                - CreateNamespace=true

    - name: Wait until Application is Healthy (basic poll)
      ansible.builtin.shell: |
        set -e
        for i in $(seq 1 60); do
          PHASE=$(kubectl -n {{ namespace }} get app {{ app_name }} -o jsonpath='{.status.health.status}' || true)
          if [ "$PHASE" = "Healthy" ]; then
            echo "Application is Healthy"
            exit 0
          fi
          sleep 5
        done
        echo "Timeout waiting for Healthy"; exit 1
      args:
        executable: /bin/bash

---

Troubleshooting (what we fixed today)

• 404 from Traefik: Ingress pointed to a non-existent service (argocd-argocd-server). FIX: route to argocd-server on port 80 (or port name http) and ensure ingressClassName: traefik.
• TLS secret mismatch: Ingress expected argocd-server-tls but we created argocd-tls. FIX: set spec.tls[0].secretName: argocd-tls.
• Missing entrypoints/TLS: Traefik v3 typically needs router.entrypoints=websecure and router.tls=true. We added both as annotations.
• Permission denied on mkcert key (localhost): When mkcert was run under sudo, the key was root-owned. FIX: use Ansible slurp with become or chown/chmod 600 locally.
• “repo not found” in Helm: Add the repo and update: helm repo add argo https://argoproj.github.io/argo-helm && helm repo update.

What’s next: Wire Argo CD to your own private repo (SSH deploy key), introduce Projects/RBAC, and manage multi-env (dev/stage/prod) via Helm values or Kustomize overlays.