HomeLab – Mkcert TLS, Helm Scale & Rollback – Day 8

Goal: Issue a locally trusted TLS cert using mkcert, rotate the Kubernetes TLS Secret for nginx.apps.lan, then practice Helm scale and rollback — fully automated by Ansible. Control host: fullstacklab.site (user stackadmin). kubectl and helm run on the k3s-master.

---

Step 1 — Issue a trusted certificate with mkcert (on the control host)

We install mkcert locally (APT or binary fallback), install its local Root CA into the system trust store, and mint a leaf cert for nginx.apps.lan. Files are stored under ~/.mkcert/nginx-apps-lan/.

---
# ansible/mkcert_local_and_secret.yml  (Play 1/2)
- name: Install mkcert locally and mint a trusted cert for nginx.apps.lan
  hosts: localhost
  connection: local
  gather_facts: false
  become: true
  vars:
    cert_dir: "{{ lookup('env','HOME') }}/.mkcert/nginx-apps-lan"
    cn: "nginx.apps.lan"
    mkcert_version: "v1.4.4"
    mkcert_bin_path: "/usr/local/bin/mkcert"
  environment:
    DEBIAN_FRONTEND: noninteractive

  pre_tasks:
    - name: Update apt cache
      ansible.builtin.apt:
        update_cache: yes
        cache_valid_time: 1800

  tasks:
    - name: Install prerequisites
      ansible.builtin.apt:
        name:
          - ca-certificates
          - libnss3-tools
          - curl
        state: present

    - name: Try install mkcert via apt (if available)
      ansible.builtin.apt:
        name: mkcert
        state: present
      register: apt_mkcert
      failed_when: false

    - name: Determine arch for mkcert binary
      ansible.builtin.shell: uname -m
      register: arch
      changed_when: false

    - name: Map arch to release filename
      ansible.builtin.set_fact:
        mkcert_asset: >-
          {{ 'mkcert-' + mkcert_version + '-linux-amd64' if arch.stdout == 'x86_64'
             else 'mkcert-' + mkcert_version + '-linux-arm64' if arch.stdout == 'aarch64'
             else 'mkcert-' + mkcert_version + '-linux-arm' if arch.stdout == 'armv7l'
             else '' }}

    - name: Download mkcert binary (fallback when apt install failed)
      when: apt_mkcert is failed or apt_mkcert is skipped or apt_mkcert is not defined
      block:
        - name: Fail if arch unsupported
          ansible.builtin.fail:
            msg: "Unsupported architecture: {{ arch.stdout }}"
          when: mkcert_asset == ''

        - name: Fetch mkcert {{ mkcert_version }}
          ansible.builtin.get_url:
            url: "https://github.com/FiloSottile/mkcert/releases/download/{{ mkcert_version }}/{{ mkcert_asset }}"
            dest: "/tmp/mkcert"
            mode: "0755"

        - name: Install mkcert binary to /usr/local/bin
          ansible.builtin.copy:
            src: "/tmp/mkcert"
            dest: "{{ mkcert_bin_path }}"
            mode: "0755"

    - name: Ensure cert dir exists
      ansible.builtin.file:
        path: "{{ cert_dir }}"
        state: directory
        mode: "0755"

    - name: Install local Root CA into system trust store
      ansible.builtin.command: mkcert -install
      changed_when: false

    - name: Mint leaf cert for {{ cn }}
      ansible.builtin.command: >
        mkcert -cert-file {{ cert_dir }}/{{ cn }}.crt
               -key-file {{ cert_dir }}/{{ cn }}.key
               {{ cn }}
      args:
        creates: "{{ cert_dir }}/{{ cn }}.crt"

Step 2 — Rotate the Kubernetes TLS Secret with mkcert (on the master)

We copy the new CRT/KEY to the master and apply/update the nginx-tls Secret in the apps namespace. Traefik will serve HTTPS using this secret.

---
# ansible/mkcert_local_and_secret.yml  (Play 2/2)
- name: Update K8s TLS secret on k3s-master using mkcert output
  hosts: k3s_master
  become: true
  vars:
    kubeconfig: "/etc/rancher/k3s/k3s.yaml"
    namespace: "apps"
    secret_name: "nginx-tls"
    cn: "nginx.apps.lan"
    local_cert_dir: "{{ lookup('env','HOME') }}/.mkcert/nginx-apps-lan"
    remote_staging_dir: "/tmp/mkcert-nginx-apps-lan"
  environment:
    KUBECONFIG: "{{ kubeconfig }}"
  tasks:
    - name: Create remote staging dir
      ansible.builtin.file:
        path: "{{ remote_staging_dir }}"
        state: directory
        mode: "0755"

    - name: Copy CRT to master
      ansible.builtin.copy:
        src: "{{ local_cert_dir }}/{{ cn }}.crt"
        dest: "{{ remote_staging_dir }}/{{ cn }}.crt"
        mode: "0644"

    - name: Copy KEY to master
      ansible.builtin.copy:
        src: "{{ local_cert_dir }}/{{ cn }}.key"
        dest: "{{ remote_staging_dir }}/{{ cn }}.key"
        mode: "0600"

    - name: Create/Apply TLS secret (idempotent)
      ansible.builtin.shell: |
        kubectl -n {{ namespace }} create secret tls {{ secret_name }} \
          --cert={{ remote_staging_dir }}/{{ cn }}.crt \
          --key={{ remote_staging_dir }}/{{ cn }}.key \
          --dry-run=client -o yaml | kubectl apply -f -

Step 3 — Helm scale to 2 replicas (and keep TLS & middleware)

We pin image.tag=latest to avoid legacy Bitnami tags that may disappear, keep TLS & Traefik middleware, and wait for a green rollout. On failure, we collect diagnostics automatically.

---
# ansible/helm_scale_and_rollback.yml
- name: Helm scale/upgrade web-nginx (with pinned image tag) and verify rollout
  hosts: k3s_master
  become: true
  vars:
    kubeconfig: "/etc/rancher/k3s/k3s.yaml"
    namespace: "apps"
    release_name: "web"
    ingress_host: "nginx.apps.lan"
    tls_secret: "nginx-tls"
    replicas: 2
    image_repo: ""
    image_tag: "latest"
    values_file: "/tmp/web-values.yaml"
  environment:
    KUBECONFIG: "{{ kubeconfig }}"

  tasks:
    - name: Ensure namespace exists (idempotent)
      ansible.builtin.command: kubectl apply -f -
      args:
        stdin: |
          apiVersion: v1
          kind: Namespace
          metadata:
            name: {{ namespace }}

    - name: Render values.yaml (replicas + image pin + ingress/TLS + Traefik middleware)
      ansible.builtin.copy:
        dest: "{{ values_file }}"
        mode: "0644"
        content: |
          replicaCount: {{ replicas }}
          {% if image_repo %}
          image:
            repository: {{ image_repo }}
            tag: {{ image_tag }}
          {% else %}
          image:
            tag: {{ image_tag }}
          {% endif %}
          service:
            type: ClusterIP
          ingress:
            enabled: true
            ingressClassName: traefik
            hostname: {{ ingress_host }}
            tls: true
            extraTls:
              - hosts: [{{ ingress_host }}]
                secretName: {{ tls_secret }}
            annotations:
              "traefik.ingress.kubernetes.io/router.middlewares": "{{ namespace }}-https-redirect@kubernetescrd"

    - block:
        - name: Helm upgrade --install with values
          ansible.builtin.command: >
            helm upgrade --install {{ release_name }} bitnami/nginx -n {{ namespace }}
            -f {{ values_file }}

        - name: Wait for deployment to become Available
          ansible.builtin.command: kubectl -n {{ namespace }} rollout status deploy/{{ release_name }}-nginx --timeout=300s

        - name: Show deploy/svc/ing after success
          ansible.builtin.command: kubectl -n {{ namespace }} get deploy,svc,ingress -o wide
          register: gsummary
          changed_when: false

        - name: Print summary
          ansible.builtin.debug:
            var: gsummary.stdout

      rescue:
        - name: Describe deployment on failure
          ansible.builtin.command: kubectl -n {{ namespace }} describe deploy/{{ release_name }}-nginx
          register: depdesc
          changed_when: false
        - ansible.builtin.debug:
            var: depdesc.stdout

        - name: List ReplicaSets
          ansible.builtin.command: kubectl -n {{ namespace }} get rs -l app.kubernetes.io/name=nginx -o wide
          register: rslist
          changed_when: false
        - ansible.builtin.debug:
            var: rslist.stdout

        - name: List pods + status
          ansible.builtin.command: kubectl -n {{ namespace }} get pods -l app.kubernetes.io/name=nginx -o wide
          register: pods
          changed_when: false
        - ansible.builtin.debug:
            var: pods.stdout

        - name: Recent events
          ansible.builtin.command: kubectl -n {{ namespace }} get events --sort-by=.lastTimestamp | tail -n 50
          register: ev
          changed_when: false
        - ansible.builtin.debug:
            var: ev.stdout

        - name: Fail explicitly with hint
          ansible.builtin.fail:
            msg: >
              Deployment did not become Available. Check image repo/tag (set image_tag=latest or image_repo=bitnamilegacy/nginx),
              nodes readiness/capacity, or Ingress/TLS configuration. See describe/pods/events above.

- name: (Optional) Show Helm history and (optionally) rollback
  hosts: k3s_master
  become: true
  vars:
    kubeconfig: "/etc/rancher/k3s/k3s.yaml"
    namespace: "apps"
    release_name: "web"
    do_rollback: false
    target_revision: ""
  environment:
    KUBECONFIG: "{{ kubeconfig }}"
  tasks:
    - name: Show Helm history
      ansible.builtin.command: helm -n {{ namespace }} history {{ release_name }}
      register: hist
      changed_when: false
    - ansible.builtin.debug:
        var: hist.stdout

    - name: Roll back to specific revision (if requested)
      when: do_rollback | bool and (target_revision | length > 0)
      block:
        - name: Helm rollback
          ansible.builtin.command: helm -n {{ namespace }} rollback {{ release_name }} {{ target_revision }}
        - name: Wait for deployment after rollback
          ansible.builtin.command: kubectl -n {{ namespace }} rollout status deploy/{{ release_name }}-nginx --timeout=300s

Step 4 — Validate from the control host (trusted HTTPS)

---
# ansible/https_validate_from_host.yml
- name: Validate HTTPS from control host
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Ensure /etc/hosts maps nginx.apps.lan to master (example: 172.16.9.131)
      become: true
      ansible.builtin.lineinfile:
        path: /etc/hosts
        create: true
        state: present
        regexp: '^\S+\s+nginx\.apps\.lan\s*$'
        line: "172.16.9.131 nginx.apps.lan"

    - name: Verify HTTPS (should work without -k when mkcert CA is trusted)
      ansible.builtin.command: curl -I https://nginx.apps.lan/
      register: head
      changed_when: false

    - name: Show response headers
      ansible.builtin.debug:
        var: head.stdout

---

Troubleshooting (what we fixed today)

• Helm rollout deadline exceeded: One ReplicaSet used an old Bitnami tag (1.29.1-debian-12-r0) that no longer pulls. Solution: pin image.tag=latest (or legacy repo/tag) and re-run Helm upgrade; the failing RS scaled to 0.
• mkcert via snap not found: There is no snap package. We used APT if available, otherwise downloaded the binary from GitHub releases.
• APT lock or long wait: Use apt with cache update and (if needed) a short lock wait. If it hangs, run sudo dpkg --configure -a and sudo apt -f install manually to fix.
• Control host pointed to the wrong IP: Ingress showed 172.16.9.130/131/132 but /etc/hosts still mapped nginx.apps.lan to 192.168.56.10. Fix: update /etc/hosts to the correct node IP (e.g., 172.16.9.131).
• Still seeing self-signed certificate: The nginx-tls Secret still contained the old self-signed cert. Replace it with mkcert CRT/KEY (kubectl create secret tls --dry-run=client | kubectl apply -f -) and, if needed, rollout restart deploy/traefik.
• scp: Permission denied on key file: ~/.mkcert/… was owned by root (mkcert run under sudo). Fix ownership (sudo chown -R $USER:$USER ~/.mkcert) and set chmod 600 on the key before copying.
• curl still complains about trust: Install mkcert Root CA into the system trust store on the control host:

  sudo cp "$(mkcert -CAROOT)/rootCA.pem" /usr/local/share/ca-certificates/mkcert-rootCA.crt
  sudo update-ca-certificates -v

  Then re-try curl -I https://nginx.apps.lan/.

---

What’s next: push the mkcert workflow into CI (optional), template a custom NGINX index via ConfigMap, and practice a clean helm rollback after a controlled failure.