HomeLab – Moving NGINX TLS from a Manual Secret to cert-manager (with mkcert CA) – Day 14

DevOps, Homelab For Devops, Linux

Today we finished the transition from a manually managed TLS secret to an automated, GitOps-friendly setup driven by cert-manager. Our NGINX ingresses now receive certificates issued by a ClusterIssuer that trusts our local mkcert CA. Below you’ll find the full Ansible playbook we used, plus the exact verification and troubleshooting commands that helped us get to green.

What we set out to do

  • Install/upgrade cert-manager (including CRDs) via Helm.
  • Import the local mkcert CA and create a ClusterIssuer.
  • Define a Certificate for nginx.apps.lan that writes to the existing secret apps/nginx-tls.
  • Annotate NGINX ingresses so cert-manager manages their TLS automatically.
  • Verify we’re really using a cert issued by the mkcert CA (and retire the manual one).

The Playbook (Ansible)

File: ansible/day14_cert_manager_nginx.yml

---
- name: Day 14 | cert-manager + mkcert CA + NGINX TLS
  hosts: k3s-master
  gather_facts: false

  vars:
    kubeconfig_path: /etc/rancher/k3s/k3s.yaml

    # --- cert-manager Helm chart sources/versions ---
    cm_namespace: cert-manager
    cm_repo_name: jetstack
    cm_repo_url: https://charts.jetstack.io
    cm_chart: jetstack/cert-manager
    cm_version: v1.15.3        # pick a recent, known-good version
    cm_values: {}              # keep defaults unless you need tweaks

    # --- mkcert ClusterIssuer (self-signed CA for homelab) ---
    issuer_name: mkcert-cluster-issuer
    issuer_kind: ClusterIssuer
    mkcert_ca_secret_ns: cert-manager
    mkcert_ca_secret_name: mkcert-ca-secret

    # Paths to your mkcert CA, already present on the control host
    mkcert_ca_crt_path: /usr/local/share/ca-certificates/mkcert-rootCA.crt
    mkcert_ca_key_path: "{{ lookup('env', 'MKCERT_CA_KEY') | default('/usr/local/share/ca-certificates/mkcert-rootCA.key', true) }}"
    # ^ The private key path must exist on the control host. For mkcert, export MKCERT_CA_KEY if it lives elsewhere.

    # --- Target app / domain ---
    app_ns: apps
    domain_name: nginx.apps.lan
    tls_secret_name: nginx-tls   # we keep the same Secret name (cert-manager will own it)

    # Ingress candidates to annotate (adjust to your repo/state)
    ingress_list:
      - { ns: "apps", name: "web-gitops-nginx" }
      - { ns: "apps", name: "web-nginx-gitops" }

  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"

  tasks:

    - name: Ensure cert-manager namespace exists
      command: kubectl get ns {{ cm_namespace }}
      register: ns_cm
      failed_when: false
      changed_when: false

    - name: Create cert-manager namespace (if missing)
      command: kubectl create ns {{ cm_namespace }}
      when: ns_cm.rc != 0

    - name: Add Helm repo (jetstack)
      command: helm repo add {{ cm_repo_name }} {{ cm_repo_url }}
      register: addrepo
      failed_when: false
      changed_when: "'has been added' in (addrepo.stdout | default('')) or 'has been added' in (addrepo.stderr | default(''))"

    - name: Helm repo update
      command: helm repo update
      changed_when: false

    - name: Install/upgrade cert-manager (with CRDs)
      command: >
        helm upgrade --install cert-manager {{ cm_chart }}
        --version {{ cm_version }}
        --namespace {{ cm_namespace }}
        --set installCRDs=true
      register: cm_install
      changed_when: "'installed' in (cm_install.stdout | lower) or 'upgraded' in (cm_install.stdout | lower) or cm_install.rc == 0"

    - name: Wait for cert-manager rollout
      shell: |
        set -euo pipefail
        kubectl -n {{ cm_namespace }} rollout status deploy/cert-manager --timeout=180s
        kubectl -n {{ cm_namespace }} rollout status deploy/cert-manager-webhook --timeout=180s
        kubectl -n {{ cm_namespace }} rollout status deploy/cert-manager-cainjector --timeout=180s
      args:
        executable: /bin/bash

    # --- Import mkcert CA into the cluster as a secret that ClusterIssuer will use ---
    - name: Verify mkcert CA files exist on control host
      delegate_to: localhost
      become: false
      stat:
        path: "{{ item }}"
      register: mkcert_files
      loop:
        - "{{ mkcert_ca_crt_path }}"
        - "{{ mkcert_ca_key_path }}"

    - name: Fail if mkcert CA files are missing
      when: item.stat.exists is not defined or not item.stat.exists
      fail:
        msg: "Missing mkcert CA piece: {{ item.stat.path }}. Provide MKCERT_CA_KEY env var or fix paths."
      loop: "{{ mkcert_files.results }}"

    - name: Create/Update mkcert CA secret for ClusterIssuer
      shell: |
        set -euo pipefail
        kubectl -n {{ mkcert_ca_secret_ns }} delete secret {{ mkcert_ca_secret_name }} --ignore-not-found
        kubectl -n {{ mkcert_ca_secret_ns }} create secret tls {{ mkcert_ca_secret_name }} \
          --cert={{ mkcert_ca_crt_path }} \
          --key={{ mkcert_ca_key_path }}
      args:
        executable: /bin/bash

    - name: Apply ClusterIssuer (mkcert)
      copy:
        dest: /tmp/clusterissuer.yaml
        mode: '0644'
        content: |
          apiVersion: cert-manager.io/v1
          kind: ClusterIssuer
          metadata:
            name: {{ issuer_name }}
          spec:
            ca:
              secretName: {{ mkcert_ca_secret_name }}
      register: ci_file

    - name: kubectl apply ClusterIssuer
      command: kubectl apply -f /tmp/clusterissuer.yaml

    - name: Wait for ClusterIssuer to be ready (best-effort)
      shell: |
        set -euo pipefail
        for i in $(seq 1 30); do
          ok=$(kubectl get clusterissuer {{ issuer_name }} -o json 2>/dev/null \
               | jq -r '.status.conditions[]? | select(.type=="Ready") | .status' || true)
          if [ "$ok" = "True" ]; then
            echo "ClusterIssuer Ready"
            exit 0
          fi
          sleep 3
        done
        echo "ClusterIssuer not ready in time" >&2
        exit 1
      args:
        executable: /bin/bash
      register: issuer_ready
      failed_when: false

    # --- Create Certificate object that will own apps/nginx-tls ---
    - name: Apply Certificate for {{ domain_name }}
      copy:
        dest: /tmp/certificate.yaml
        mode: '0644'
        content: |
          apiVersion: cert-manager.io/v1
          kind: Certificate
          metadata:
            name: {{ tls_secret_name }}
            namespace: {{ app_ns }}
          spec:
            secretName: {{ tls_secret_name }}
            dnsNames:
              - {{ domain_name }}
            issuerRef:
              name: {{ issuer_name }}
              kind: ClusterIssuer
      register: cert_file

    - name: kubectl apply Certificate
      command: kubectl apply -f /tmp/certificate.yaml

    # --- Annotate ingresses so cert-manager manages TLS on the same secretName ---
    - name: Annotate ingresses with issuer (idempotent)
      shell: |
        set -euo pipefail
        for x in {{ ingress_list | to_json }}; do
          ns=$(echo "$x" | jq -r '.ns')
          name=$(echo "$x" | jq -r '.name')
          if kubectl -n "$ns" get ingress "$name" >/dev/null 2>&1; then
            kubectl -n "$ns" annotate ingress "$name" cert-manager.io/cluster-issuer={{ issuer_name }} --overwrite
            kubectl -n "$ns" patch ingress "$name" --type merge -p '{"spec":{"tls":[{"hosts":["{{ domain_name }}"],"secretName":"{{ tls_secret_name }}"}]}}'
          fi
        done
      args:
        executable: /bin/bash

    # --- Wait for Certificate to become Ready ---
    - name: Wait until Certificate is Ready
      shell: |
        set -euo pipefail
        for i in $(seq 1 60); do
          ready=$(kubectl -n {{ app_ns }} get certificate {{ tls_secret_name }} -o json \
                  | jq -r '.status.conditions[]? | select(.type=="Ready") | .status' 2>/dev/null || true)
          if [ "$ready" = "True" ]; then
            echo "Certificate Ready"
            exit 0
          fi
          sleep 3
        done
        echo "Certificate not Ready in time" >&2
        exit 1
      args:
        executable: /bin/bash

    # --- Verification: print issuer + validity of tls.crt ---
    - name: Verify resulting certificate chain
      shell: |
        set -euo pipefail
        kubectl -n {{ app_ns }} get secret {{ tls_secret_name }} -o jsonpath='{.data.tls\.crt}' \
        | base64 -d | openssl x509 -noout -subject -issuer -dates
      args:
        executable: /bin/bash
      register: crt_info
      changed_when: false

    - name: Show openssl result
      debug:
        msg: "{{ crt_info.stdout.split('\n') }}"

How we verified the switch

Key checks we ran after the playbook finished:

# 1) Certificate object is Ready
kubectl -n apps get certificate nginx-tls -o wide

# 2) The Secret is now owned by cert-manager and signed by mkcert CA
kubectl -n apps get secret nginx-tls -o yaml | sed -n '1,120p'

# 3) Confirm issuer & validity window of the actual certificate bytes
kubectl -n apps get secret nginx-tls -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -subject -issuer -dates

# 4) Both ingresses point to the same Secret (nginx-tls)
for ing in web-gitops-nginx web-nginx-gitops; do
  echo -n "$ing -> "
  kubectl -n apps get ingress "$ing" -o jsonpath='{.spec.tls[*].secretName}'; echo
done

# 5) End-to-end test from client using mkcert root CA bundle
curl -I https://nginx.apps.lan --cacert "$(mkcert -CAROOT)/rootCA.pem"

Troubleshooting we actually used

We hit one transient state: “IncorrectCertificate: Secret was issued for "nginx-apps-cert"… you might have two conflicting Certificates pointing to the same secret.” The fix is to ensure there’s only a single Certificate object that owns apps/nginx-tls and let cert-manager re-issue it.

# List all certificates across namespaces
kubectl get certificate -A -o wide

# Describe the problematic Certificate and inspect Events and conditions
kubectl -n apps describe certificate nginx-tls | sed -n '/Events/,$p'

# If a stale Secret is causing confusion, you can force a fresh issuance:
# (cert-manager will recreate the secret based on the Certificate spec)
kubectl -n apps delete secret nginx-tls

# Re-check readiness after a short while
kubectl -n apps get certificate nginx-tls -o wide

# cert-manager controller logs (if deeper debugging is needed)
kubectl -n cert-manager logs deploy/cert-manager --tail=200

# Confirm the Ingress annotations & TLS stanza are correct
kubectl -n apps get ingress web-gitops-nginx -o yaml | sed -n '1,200p'
kubectl -n apps get ingress web-nginx-gitops -o yaml | sed -n '1,200p'

Result

Both NGINX ingresses now reference apps/nginx-tls, and the underlying certificate is issued and renewed by cert-manager via our mkcert ClusterIssuer. That means:

  • No more hand-rolled TLS secrets.
  • Automatic renewals before expiry.
  • Predictable, declarative configuration you can manage in Git.

Next, we’ll tighten our GitOps loop by committing the ClusterIssuer, Certificate and Ingress manifests to the repo so the entire TLS story is fully declarative from source control.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.