From f259285f6e9748f14c1a222d05c9e56c8fc8396d Mon Sep 17 00:00:00 2001 From: Nikholas Pcenicni <82239765+nikpcenicni@users.noreply.github.com> Date: Wed, 1 Apr 2026 01:55:41 -0400 Subject: [PATCH] Enhance Argo CD integration by adding support for a bootstrap root application. Update `group_vars/all.yml` and role defaults to include `noble_argocd_apply_bootstrap_root_application`. Modify tasks to apply the bootstrap application conditionally. Revise documentation to clarify the GitOps workflow and the relationship between the core platform and optional applications. Remove outdated references and streamline the README for better user guidance. --- ansible/README.md | 2 +- ansible/group_vars/all.yml | 2 + ansible/roles/noble_argocd/defaults/main.yml | 2 + ansible/roles/noble_argocd/tasks/main.yml | 12 +++++ .../roles/noble_post_deploy/tasks/main.yml | 7 +-- clusters/noble/apps/README.md | 4 +- clusters/noble/apps/kustomization.yaml | 3 -- clusters/noble/bootstrap/argocd/README.md | 51 ++++++++++++++++--- .../argocd/app-of-apps/kustomization.yaml | 6 +++ .../argocd/bootstrap-root-application.yaml | 29 +++++++++++ .../bootstrap/argocd/root-application.yaml | 6 ++- clusters/noble/bootstrap/kustomization.yaml | 5 +- docs/architecture.md | 2 +- talos/CLUSTER-BUILD.md | 4 +- 14 files changed, 112 insertions(+), 23 deletions(-) create mode 100644 clusters/noble/bootstrap/argocd/app-of-apps/kustomization.yaml create mode 100644 clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml diff --git a/ansible/README.md b/ansible/README.md index bbef493..cb95661 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -79,7 +79,7 @@ ansible-playbook playbooks/noble.yml --tags velero -e noble_velero_install=true ### Variables — `group_vars/all.yml` and role defaults -- **`group_vars/all.yml`:** **`noble_newt_install`**, **`noble_velero_install`**, **`noble_cert_manager_require_cloudflare_secret`**, **`noble_argocd_apply_root_application`**, **`noble_k8s_api_server_override`**, **`noble_k8s_api_server_auto_fallback`**, **`noble_k8s_api_server_fallback`**, **`noble_skip_k8s_health_check`** +- **`group_vars/all.yml`:** **`noble_newt_install`**, **`noble_velero_install`**, **`noble_cert_manager_require_cloudflare_secret`**, **`noble_argocd_apply_root_application`**, **`noble_argocd_apply_bootstrap_root_application`**, **`noble_k8s_api_server_override`**, **`noble_k8s_api_server_auto_fallback`**, **`noble_k8s_api_server_fallback`**, **`noble_skip_k8s_health_check`** - **`roles/noble_platform/defaults/main.yml`:** **`noble_apply_sops_secrets`**, **`noble_sops_age_key_file`** (SOPS secrets under **`clusters/noble/secrets/`**) ## Roles diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all.yml index 4177088..bf214ad 100644 --- a/ansible/group_vars/all.yml +++ b/ansible/group_vars/all.yml @@ -24,3 +24,5 @@ noble_velero_install: false # Argo CD — apply app-of-apps root Application (clusters/noble/bootstrap/argocd/root-application.yaml). Set false to skip. noble_argocd_apply_root_application: true +# Bootstrap kustomize in Argo (**noble-bootstrap-root** → **clusters/noble/bootstrap**). Applied with manual sync; enable automation after **noble.yml** (see **clusters/noble/bootstrap/argocd/README.md** §5). +noble_argocd_apply_bootstrap_root_application: true diff --git a/ansible/roles/noble_argocd/defaults/main.yml b/ansible/roles/noble_argocd/defaults/main.yml index 2acb1c4..48300ae 100644 --- a/ansible/roles/noble_argocd/defaults/main.yml +++ b/ansible/roles/noble_argocd/defaults/main.yml @@ -2,3 +2,5 @@ # When true, applies clusters/noble/bootstrap/argocd/root-application.yaml (app-of-apps). # Edit spec.source.repoURL in that file if your Git remote differs. noble_argocd_apply_root_application: false +# When true, applies clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml (noble-bootstrap-root; manual sync until README §5). +noble_argocd_apply_bootstrap_root_application: true diff --git a/ansible/roles/noble_argocd/tasks/main.yml b/ansible/roles/noble_argocd/tasks/main.yml index 57c0d1f..6cdf499 100644 --- a/ansible/roles/noble_argocd/tasks/main.yml +++ b/ansible/roles/noble_argocd/tasks/main.yml @@ -32,3 +32,15 @@ KUBECONFIG: "{{ noble_kubeconfig }}" when: noble_argocd_apply_root_application | default(false) | bool changed_when: true + +- name: Apply Argo CD bootstrap app-of-apps Application + ansible.builtin.command: + argv: + - kubectl + - apply + - -f + - "{{ noble_repo_root }}/clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml" + environment: + KUBECONFIG: "{{ noble_kubeconfig }}" + when: noble_argocd_apply_bootstrap_root_application | default(false) | bool + changed_when: true diff --git a/ansible/roles/noble_post_deploy/tasks/main.yml b/ansible/roles/noble_post_deploy/tasks/main.yml index 8b3e350..8ee779e 100644 --- a/ansible/roles/noble_post_deploy/tasks/main.yml +++ b/ansible/roles/noble_post_deploy/tasks/main.yml @@ -9,6 +9,7 @@ - name: Argo CD optional root Application (empty app-of-apps) ansible.builtin.debug: msg: >- - App-of-apps: noble.yml applies root-application.yaml when noble_argocd_apply_root_application is true - (group_vars/all.yml). Otherwise: kubectl apply -f clusters/noble/bootstrap/argocd/root-application.yaml - after editing spec.source.repoURL. Core platform is Ansible — see clusters/noble/apps/README.md + App-of-apps: noble.yml applies root-application.yaml when noble_argocd_apply_root_application is true; + bootstrap-root-application.yaml when noble_argocd_apply_bootstrap_root_application is true (group_vars/all.yml). + noble-bootstrap-root uses manual sync until you enable automation after the playbook — + clusters/noble/bootstrap/argocd/README.md §5. See clusters/noble/apps/README.md and that README. diff --git a/clusters/noble/apps/README.md b/clusters/noble/apps/README.md index 57b1370..2e5d954 100644 --- a/clusters/noble/apps/README.md +++ b/clusters/noble/apps/README.md @@ -2,6 +2,6 @@ **Base cluster configuration** (CNI, MetalLB, ingress, cert-manager, storage, observability stack, policy, SOPS secrets path, etc.) is installed by **`ansible/playbooks/noble.yml`** from **`clusters/noble/bootstrap/`** — not from here. -**`noble-root`** (`clusters/noble/bootstrap/argocd/root-application.yaml`) points at **`clusters/noble/apps`**. Add **`Application`** manifests (and optional **`AppProject`** definitions) under this directory only for workloads that are additive and do not subsume the Ansible-managed platform. +**`noble-root`** (`clusters/noble/bootstrap/argocd/root-application.yaml`) points at **`clusters/noble/apps`**. Add **`Application`** manifests (and optional **`AppProject`** definitions) under this directory only for workloads that are additive and do not subsume the core platform. -For an app-of-apps pattern, use a second-level **`Application`** that syncs a subdirectory (for example **`optional/`**) containing leaf **`Application`** resources. +Bootstrap kustomize (namespaces, static YAML, leaf **`Application`**s) lives in **`clusters/noble/bootstrap/`** and is tracked by **`noble-bootstrap-root`** — enable automated sync for that app only after **`noble.yml`** completes (**`clusters/noble/bootstrap/argocd/README.md`** §5). Put Helm **`Application`** migrations under **`clusters/noble/bootstrap/argocd/app-of-apps/`**. diff --git a/clusters/noble/apps/kustomization.yaml b/clusters/noble/apps/kustomization.yaml index 9f58d5a..d0d72de 100644 --- a/clusters/noble/apps/kustomization.yaml +++ b/clusters/noble/apps/kustomization.yaml @@ -5,6 +5,3 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - homepage/application.yaml - - eclipse-che/application-devworkspace.yaml - - eclipse-che/application-operator.yaml - - eclipse-che/application-checluster.yaml diff --git a/clusters/noble/bootstrap/argocd/README.md b/clusters/noble/bootstrap/argocd/README.md index aa6338f..0190815 100644 --- a/clusters/noble/bootstrap/argocd/README.md +++ b/clusters/noble/bootstrap/argocd/README.md @@ -50,21 +50,56 @@ helm upgrade --install argocd argo/argo-cd -n argocd --create-namespace \ Use **Settings → Repositories** in the UI, or `argocd repo add` / a `Secret` of type `repository`. -## 4. App-of-apps (optional GitOps only) +## 4. App-of-apps (GitOps) -Bootstrap **platform** workloads (CNI, ingress, cert-manager, Kyverno, observability, etc.) are installed by -**`ansible/playbooks/noble.yml`** from **`clusters/noble/bootstrap/`** — not by Argo. **`clusters/noble/apps/kustomization.yaml`** is empty by default. +**Ansible** (`ansible/playbooks/noble.yml`) performs the **initial** install: Helm releases and **`kubectl apply -k clusters/noble/bootstrap`**. **Argo** then tracks the same git paths for ongoing reconciliation. -1. Edit **`root-application.yaml`**: set **`repoURL`** and **`targetRevision`** to this repository. The **`resources-finalizer.argocd.argoproj.io/background`** finalizer uses Argo’s path-qualified form so **`kubectl apply`** does not warn about finalizer names. -2. When you want Argo to manage specific apps, add **`Application`** manifests under **`clusters/noble/apps/`** (see **`clusters/noble/apps/README.md`**). -3. Apply the root: +1. Edit **`root-application.yaml`** and **`bootstrap-root-application.yaml`**: set **`repoURL`** and **`targetRevision`**. The **`resources-finalizer.argocd.argoproj.io/background`** finalizer uses Argo’s path-qualified form so **`kubectl apply`** does not warn about finalizer names. +2. Optional add-on apps: add **`Application`** manifests under **`clusters/noble/apps/`** (see **`clusters/noble/apps/README.md`**). +3. **Bootstrap kustomize** (namespaces, datasource, leaf **`Application`**s under **`argocd/app-of-apps/`**, etc.): **`noble-bootstrap-root`** syncs **`clusters/noble/bootstrap`**. It is created with **manual** sync only so Argo does not apply changes while **`noble.yml`** is still running. + + **`ansible/playbooks/noble.yml`** (role **`noble_argocd`**) applies both roots when **`noble_argocd_apply_root_application`** / **`noble_argocd_apply_bootstrap_root_application`** are true in **`ansible/group_vars/all.yml`**. ```bash kubectl apply -f clusters/noble/bootstrap/argocd/root-application.yaml + kubectl apply -f clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml ``` -If you migrated from GitOps-managed **`noble-platform`** / **`noble-kyverno`**, delete stale **`Application`** objects on -the cluster (see **`clusters/noble/apps/README.md`**) then re-apply the root. +If you migrated from older GitOps **`Application`** names, delete stale **`Application`** objects on the cluster (see **`clusters/noble/apps/README.md`**) then re-apply the roots. + +## 5. After Ansible: enable automated sync for **noble-bootstrap-root** + +Do this only after **`ansible-playbook playbooks/noble.yml`** has finished successfully (including **`noble_platform`** `kubectl apply -k` and any Helm stages you rely on). Until then, leave **manual** sync so Argo does not fight the playbook. + +**Required steps** + +1. Confirm the cluster matches git for kustomize output (optional): `kubectl kustomize clusters/noble/bootstrap | kubectl diff -f -` or inspect resources in the UI. +2. Register the git repo in Argo if you have not already (**§3**). +3. **Refresh** the app so Argo compares **`clusters/noble/bootstrap`** to the cluster: Argo UI → **noble-bootstrap-root** → **Refresh**, or: + + ```bash + argocd app get noble-bootstrap-root --refresh + ``` + +4. **Enable automated sync** (prune + self-heal), preserving **`CreateNamespace`**, using any one of: + + **kubectl** + + ```bash + kubectl patch application noble-bootstrap-root -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true},"syncOptions":["CreateNamespace=true"]}}}' + ``` + + **argocd** CLI (logged in) + + ```bash + argocd app set noble-bootstrap-root --sync-policy automated --auto-prune --self-heal + ``` + + **UI:** open **noble-bootstrap-root** → **App Details** → enable **AUTO-SYNC** (and **Prune** / **Self Heal** if shown). + +5. Trigger a sync if the app does not go green immediately: **Sync** in the UI, or `argocd app sync noble-bootstrap-root`. + +After this, **git** is the source of truth for everything under **`clusters/noble/bootstrap/kustomization.yaml`** (including **`argocd/app-of-apps/`**). Helm-managed platform components remain whatever Ansible last installed until you model them as Argo **`Application`**s under **`app-of-apps/`** and stop installing them from Ansible. ## Versions diff --git a/clusters/noble/bootstrap/argocd/app-of-apps/kustomization.yaml b/clusters/noble/bootstrap/argocd/app-of-apps/kustomization.yaml new file mode 100644 index 0000000..602653c --- /dev/null +++ b/clusters/noble/bootstrap/argocd/app-of-apps/kustomization.yaml @@ -0,0 +1,6 @@ +# Sub-kustomization included by **clusters/noble/bootstrap/kustomization.yaml**. Leaf **Application** / +# **AppProject** resources (Helm apps you migrate off raw **helm upgrade** in Ansible). Synced with the +# rest of **clusters/noble/bootstrap** via **noble-bootstrap-root** once automated sync is enabled. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: [] diff --git a/clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml b/clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml new file mode 100644 index 0000000..f917d03 --- /dev/null +++ b/clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml @@ -0,0 +1,29 @@ +# **noble-bootstrap-root** — Kustomize app-of-apps for **clusters/noble/bootstrap** (same tree as +# **ansible/playbooks/noble.yml** → **noble_platform** `kubectl apply -k`). +# +# **Initial deploy:** Ansible is the only writer; **automated sync is off** so Argo does not reconcile +# during **noble.yml**. **After** the playbook finishes, enable automated sync (see **README.md** §5) +# so git becomes the source of truth for this kustomize output. +# +# Edit **spec.source.repoURL** / **targetRevision** for your remote. +# +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: noble-bootstrap-root + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io/background +spec: + project: default + source: + repoURL: https://gitea.pcenicni.ca/gsdavidp/home-server.git + targetRevision: HEAD + path: clusters/noble/bootstrap + destination: + server: https://kubernetes.default.svc + namespace: argocd + # Manual sync until you enable automation after Ansible (see README.md §5). + syncPolicy: + syncOptions: + - CreateNamespace=true diff --git a/clusters/noble/bootstrap/argocd/root-application.yaml b/clusters/noble/bootstrap/argocd/root-application.yaml index 7fd72e4..2f466f0 100644 --- a/clusters/noble/bootstrap/argocd/root-application.yaml +++ b/clusters/noble/bootstrap/argocd/root-application.yaml @@ -3,8 +3,10 @@ # 1. Set spec.source.repoURL (and targetRevision — **HEAD** tracks the remote default branch) to this repo. # 2. kubectl apply -f clusters/noble/bootstrap/argocd/root-application.yaml # -# **clusters/noble/apps** holds optional **Application** manifests. Core platform is installed by -# **ansible/playbooks/noble.yml** from **clusters/noble/bootstrap/**. +# **clusters/noble/apps** holds optional **Application** manifests. Core platform Helm + kustomize is +# installed by **ansible/playbooks/noble.yml** from **clusters/noble/bootstrap/**. **bootstrap-root-application.yaml** +# registers **noble-bootstrap-root** for the same kustomize tree (**manual** sync until you enable +# automation after the playbook — see **README.md** §5). # apiVersion: argoproj.io/v1alpha1 kind: Application diff --git a/clusters/noble/bootstrap/kustomization.yaml b/clusters/noble/bootstrap/kustomization.yaml index bebf821..88e7293 100644 --- a/clusters/noble/bootstrap/kustomization.yaml +++ b/clusters/noble/bootstrap/kustomization.yaml @@ -1,6 +1,8 @@ # Ansible bootstrap: plain Kustomize (namespaces + extra YAML). Helm installs are driven by # **ansible/playbooks/noble.yml** (role **noble_platform**) — avoids **kustomize --enable-helm** in-repo. -# Optional GitOps workloads live under **../apps/** (Argo **noble-root**). +# Optional GitOps: **../apps/** (Argo **noble-root**); leaf **Application**s under **argocd/app-of-apps/**. +# **noble-bootstrap-root** (Argo) uses this same path — enable automated sync only after **noble.yml** +# completes (see **argocd/README.md** §5). apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization @@ -14,3 +16,4 @@ resources: - velero/longhorn-volumesnapshotclass.yaml - headlamp/namespace.yaml - grafana-loki-datasource/loki-datasource.yaml + - argocd/app-of-apps diff --git a/docs/architecture.md b/docs/architecture.md index 59bb976..feaed55 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -214,7 +214,7 @@ See [`talos/CLUSTER-BUILD.md`](../talos/CLUSTER-BUILD.md) for the authoritative ## Narrative -The **noble** environment is a **Talos** lab cluster on **`192.168.50.0/24`** with **three control plane nodes and one worker**, schedulable workloads on control planes enabled, and the Kubernetes API exposed through **kube-vip** at **`192.168.50.230`**. **Cilium** provides the CNI after Talos bootstrap with **`cni: none`**; **MetalLB** advertises **`192.168.50.210`–`192.168.50.229`**, pinning **Argo CD** to **`192.168.50.210`** and **Traefik** to **`192.168.50.211`** for **`*.apps.noble.lab.pcenicni.dev`**. **cert-manager** issues certificates for Traefik Ingresses; **GitOps** is **Ansible-driven Helm** for the platform (**`clusters/noble/bootstrap/`**) plus optional **Argo CD** app-of-apps (**`clusters/noble/apps/`**, **`clusters/noble/bootstrap/argocd/`**). **Observability** uses **kube-prometheus-stack** in **`monitoring`**, **Loki** and **Fluent Bit** with Grafana wired via a **ConfigMap** datasource, with **Longhorn** PVCs for Prometheus, Grafana, Alertmanager, and Loki. **Secrets** in git use **SOPS** + **age** under **`clusters/noble/secrets/`**; **Kyverno** enforces **Pod Security Standards baseline** in **Audit**. **Public** access uses **Newt** to **Pangolin** with **CNAME** and Integration API steps as documented—not generic in-cluster public DNS. +The **noble** environment is a **Talos** lab cluster on **`192.168.50.0/24`** with **three control plane nodes and one worker**, schedulable workloads on control planes enabled, and the Kubernetes API exposed through **kube-vip** at **`192.168.50.230`**. **Cilium** provides the CNI after Talos bootstrap with **`cni: none`**; **MetalLB** advertises **`192.168.50.210`–`192.168.50.229`**, pinning **Argo CD** to **`192.168.50.210`** and **Traefik** to **`192.168.50.211`** for **`*.apps.noble.lab.pcenicni.dev`**. **cert-manager** issues certificates for Traefik Ingresses; **GitOps** is **Ansible** for the **initial** platform install (**`clusters/noble/bootstrap/`**), then **Argo CD** for the kustomize tree (**`noble-bootstrap-root`** → **`clusters/noble/bootstrap`**) and optional apps (**`noble-root`** → **`clusters/noble/apps/`**) once automated sync is enabled after **`noble.yml`** (see **`clusters/noble/bootstrap/argocd/README.md`** §5). **Observability** uses **kube-prometheus-stack** in **`monitoring`**, **Loki** and **Fluent Bit** with Grafana wired via a **ConfigMap** datasource, with **Longhorn** PVCs for Prometheus, Grafana, Alertmanager, and Loki. **Secrets** in git use **SOPS** + **age** under **`clusters/noble/secrets/`**; **Kyverno** enforces **Pod Security Standards baseline** in **Audit**. **Public** access uses **Newt** to **Pangolin** with **CNAME** and Integration API steps as documented—not generic in-cluster public DNS. --- diff --git a/talos/CLUSTER-BUILD.md b/talos/CLUSTER-BUILD.md index a8725bc..a7673fe 100644 --- a/talos/CLUSTER-BUILD.md +++ b/talos/CLUSTER-BUILD.md @@ -16,7 +16,7 @@ Lab stack is **up** on-cluster through **Phase D**–**F** and **Phase G** (**`t - **Traefik** Helm **39.0.6** / app **v3.6.11** — `clusters/noble/bootstrap/traefik/`; **`Service`** **`LoadBalancer`** **`EXTERNAL-IP` `192.168.50.211`**; **`IngressClass`** **`traefik`** (default). Point **`*.apps.noble.lab.pcenicni.dev`** at **`192.168.50.211`**. MetalLB pool verification was done before replacing the temporary nginx test with Traefik. - **cert-manager** Helm **v1.20.0** / app **v1.20.0** — `clusters/noble/bootstrap/cert-manager/`; **`ClusterIssuer`** **`letsencrypt-staging`** and **`letsencrypt-prod`** (**DNS-01** via **Cloudflare** for **`pcenicni.dev`**, Secret **`cloudflare-dns-api-token`** in **`cert-manager`**); ACME email **`certificates@noble.lab.pcenicni.dev`** (edit in manifests if you want a different mailbox). - **Newt** Helm **1.2.0** / app **1.10.1** — `clusters/noble/bootstrap/newt/` (**fossorial/newt**); Pangolin site tunnel — **`newt-pangolin-auth`** Secret (**`PANGOLIN_ENDPOINT`**, **`NEWT_ID`**, **`NEWT_SECRET`**). Store credentials in git with **SOPS** (`clusters/noble/secrets/newt-pangolin-auth.secret.yaml`, **`age-key.txt`**, **`.sops.yaml`**) — see **`clusters/noble/secrets/README.md`**. **Public DNS** is **not** automated with ExternalDNS: **CNAME** records at your DNS host per Pangolin’s domain instructions, plus **Integration API** for HTTP resources/targets — see **`clusters/noble/bootstrap/newt/README.md`**. LAN access to Traefik can still use **`*.apps.noble.lab.pcenicni.dev`** → **`192.168.50.211`** (split horizon / local resolver). -- **Argo CD** Helm **9.4.17** / app **v3.3.6** — `clusters/noble/bootstrap/argocd/`; **`argocd-server`** **`LoadBalancer`** **`192.168.50.210`**; app-of-apps root syncs **`clusters/noble/apps/`** (edit **`root-application.yaml`** `repoURL` before applying). +- **Argo CD** Helm **9.4.17** / app **v3.3.6** — `clusters/noble/bootstrap/argocd/`; **`argocd-server`** **`LoadBalancer`** **`192.168.50.210`**; **`noble-root`** → **`clusters/noble/apps/`**; **`noble-bootstrap-root`** → **`clusters/noble/bootstrap`** (manual sync until **`argocd/README.md`** §5 after **`noble.yml`**). Edit **`repoURL`** in both root **`Application`** files before applying. - **kube-prometheus-stack** — Helm chart **82.15.1** — `clusters/noble/bootstrap/kube-prometheus-stack/` (**namespace** `monitoring`, PSA **privileged** — **node-exporter** needs host mounts); **Longhorn** PVCs for Prometheus, Grafana, Alertmanager; **node-exporter** DaemonSet **4/4**. **Grafana Ingress:** **`https://grafana.apps.noble.lab.pcenicni.dev`** (Traefik **`ingressClassName: traefik`**, **`cert-manager.io/cluster-issuer: letsencrypt-prod`**). **Loki** datasource in Grafana: ConfigMap **`clusters/noble/bootstrap/grafana-loki-datasource/loki-datasource.yaml`** (sidecar label **`grafana_datasource: "1"`**) — not via **`grafana.additionalDataSources`** in the chart. **`helm upgrade --install` with `--wait` is silent until done** — use **`--timeout 30m`**; Grafana admin: Secret **`kube-prometheus-grafana`**, keys **`admin-user`** / **`admin-password`**. - **Loki** + **Fluent Bit** — **`grafana/loki` 6.55.0** SingleBinary + **filesystem** on **Longhorn** (`clusters/noble/bootstrap/loki/`); **`loki.auth_enabled: false`**; **`chunksCache.enabled: false`** (no memcached chunk cache). **`fluent/fluent-bit` 0.56.0** → **`loki-gateway.loki.svc:80`** (`clusters/noble/bootstrap/fluent-bit/`); **`logging`** PSA **privileged**. **Grafana Explore:** **`kubectl apply -f clusters/noble/bootstrap/grafana-loki-datasource/loki-datasource.yaml`** then **Explore → Loki** (e.g. `{job="fluent-bit"}`). - **SOPS** — cluster **`Secret`** manifests under **`clusters/noble/secrets/`** encrypted with **age** (see **`.sops.yaml`**, **`age-key.txt`** gitignored); **`noble.yml`** decrypt-applies when the private key is present. @@ -86,7 +86,7 @@ Lab stack is **up** on-cluster through **Phase D**–**F** and **Phase G** (**`t | Traefik (Helm values) | `clusters/noble/bootstrap/traefik/` — `values.yaml`, `namespace.yaml`, `README.md` | | cert-manager (Helm + ClusterIssuers) | `clusters/noble/bootstrap/cert-manager/` — `values.yaml`, `namespace.yaml`, `kustomization.yaml`, `README.md` | | Newt / Pangolin tunnel (Helm) | `clusters/noble/bootstrap/newt/` — `values.yaml`, `namespace.yaml`, `README.md` | -| Argo CD (Helm) + optional app-of-apps | `clusters/noble/bootstrap/argocd/` — `values.yaml`, `root-application.yaml`, `README.md`; optional **`Application`** tree in **`clusters/noble/apps/`** | +| Argo CD (Helm) + app-of-apps | `clusters/noble/bootstrap/argocd/` — `values.yaml`, `root-application.yaml`, `bootstrap-root-application.yaml`, `app-of-apps/`, `README.md`; **`noble-root`** syncs **`clusters/noble/apps/`**; **`noble-bootstrap-root`** syncs **`clusters/noble/bootstrap`** (enable automation after **`noble.yml`**) | | kube-prometheus-stack (Helm values) | `clusters/noble/bootstrap/kube-prometheus-stack/` — `values.yaml`, `namespace.yaml` | | Grafana Loki datasource (ConfigMap; no chart change) | `clusters/noble/bootstrap/grafana-loki-datasource/loki-datasource.yaml` | | Loki (Helm values) | `clusters/noble/bootstrap/loki/` — `values.yaml`, `namespace.yaml` |