diff --git a/ansible/README.md b/ansible/README.md index bbef493..cb95661 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -79,7 +79,7 @@ ansible-playbook playbooks/noble.yml --tags velero -e noble_velero_install=true ### Variables — `group_vars/all.yml` and role defaults -- **`group_vars/all.yml`:** **`noble_newt_install`**, **`noble_velero_install`**, **`noble_cert_manager_require_cloudflare_secret`**, **`noble_argocd_apply_root_application`**, **`noble_k8s_api_server_override`**, **`noble_k8s_api_server_auto_fallback`**, **`noble_k8s_api_server_fallback`**, **`noble_skip_k8s_health_check`** +- **`group_vars/all.yml`:** **`noble_newt_install`**, **`noble_velero_install`**, **`noble_cert_manager_require_cloudflare_secret`**, **`noble_argocd_apply_root_application`**, **`noble_argocd_apply_bootstrap_root_application`**, **`noble_k8s_api_server_override`**, **`noble_k8s_api_server_auto_fallback`**, **`noble_k8s_api_server_fallback`**, **`noble_skip_k8s_health_check`** - **`roles/noble_platform/defaults/main.yml`:** **`noble_apply_sops_secrets`**, **`noble_sops_age_key_file`** (SOPS secrets under **`clusters/noble/secrets/`**) ## Roles diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all.yml index 4177088..bf214ad 100644 --- a/ansible/group_vars/all.yml +++ b/ansible/group_vars/all.yml @@ -24,3 +24,5 @@ noble_velero_install: false # Argo CD — apply app-of-apps root Application (clusters/noble/bootstrap/argocd/root-application.yaml). Set false to skip. noble_argocd_apply_root_application: true +# Bootstrap kustomize in Argo (**noble-bootstrap-root** → **clusters/noble/bootstrap**). Applied with manual sync; enable automation after **noble.yml** (see **clusters/noble/bootstrap/argocd/README.md** §5). +noble_argocd_apply_bootstrap_root_application: true diff --git a/ansible/roles/noble_argocd/defaults/main.yml b/ansible/roles/noble_argocd/defaults/main.yml index 2acb1c4..48300ae 100644 --- a/ansible/roles/noble_argocd/defaults/main.yml +++ b/ansible/roles/noble_argocd/defaults/main.yml @@ -2,3 +2,5 @@ # When true, applies clusters/noble/bootstrap/argocd/root-application.yaml (app-of-apps). # Edit spec.source.repoURL in that file if your Git remote differs. noble_argocd_apply_root_application: false +# When true, applies clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml (noble-bootstrap-root; manual sync until README §5). +noble_argocd_apply_bootstrap_root_application: true diff --git a/ansible/roles/noble_argocd/tasks/main.yml b/ansible/roles/noble_argocd/tasks/main.yml index 57c0d1f..6cdf499 100644 --- a/ansible/roles/noble_argocd/tasks/main.yml +++ b/ansible/roles/noble_argocd/tasks/main.yml @@ -32,3 +32,15 @@ KUBECONFIG: "{{ noble_kubeconfig }}" when: noble_argocd_apply_root_application | default(false) | bool changed_when: true + +- name: Apply Argo CD bootstrap app-of-apps Application + ansible.builtin.command: + argv: + - kubectl + - apply + - -f + - "{{ noble_repo_root }}/clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml" + environment: + KUBECONFIG: "{{ noble_kubeconfig }}" + when: noble_argocd_apply_bootstrap_root_application | default(false) | bool + changed_when: true diff --git a/ansible/roles/noble_post_deploy/tasks/main.yml b/ansible/roles/noble_post_deploy/tasks/main.yml index 8b3e350..8ee779e 100644 --- a/ansible/roles/noble_post_deploy/tasks/main.yml +++ b/ansible/roles/noble_post_deploy/tasks/main.yml @@ -9,6 +9,7 @@ - name: Argo CD optional root Application (empty app-of-apps) ansible.builtin.debug: msg: >- - App-of-apps: noble.yml applies root-application.yaml when noble_argocd_apply_root_application is true - (group_vars/all.yml). Otherwise: kubectl apply -f clusters/noble/bootstrap/argocd/root-application.yaml - after editing spec.source.repoURL. Core platform is Ansible — see clusters/noble/apps/README.md + App-of-apps: noble.yml applies root-application.yaml when noble_argocd_apply_root_application is true; + bootstrap-root-application.yaml when noble_argocd_apply_bootstrap_root_application is true (group_vars/all.yml). + noble-bootstrap-root uses manual sync until you enable automation after the playbook — + clusters/noble/bootstrap/argocd/README.md §5. See clusters/noble/apps/README.md and that README. diff --git a/clusters/noble/apps/README.md b/clusters/noble/apps/README.md index 57b1370..2e5d954 100644 --- a/clusters/noble/apps/README.md +++ b/clusters/noble/apps/README.md @@ -2,6 +2,6 @@ **Base cluster configuration** (CNI, MetalLB, ingress, cert-manager, storage, observability stack, policy, SOPS secrets path, etc.) is installed by **`ansible/playbooks/noble.yml`** from **`clusters/noble/bootstrap/`** — not from here. -**`noble-root`** (`clusters/noble/bootstrap/argocd/root-application.yaml`) points at **`clusters/noble/apps`**. Add **`Application`** manifests (and optional **`AppProject`** definitions) under this directory only for workloads that are additive and do not subsume the Ansible-managed platform. +**`noble-root`** (`clusters/noble/bootstrap/argocd/root-application.yaml`) points at **`clusters/noble/apps`**. Add **`Application`** manifests (and optional **`AppProject`** definitions) under this directory only for workloads that are additive and do not subsume the core platform. -For an app-of-apps pattern, use a second-level **`Application`** that syncs a subdirectory (for example **`optional/`**) containing leaf **`Application`** resources. +Bootstrap kustomize (namespaces, static YAML, leaf **`Application`**s) lives in **`clusters/noble/bootstrap/`** and is tracked by **`noble-bootstrap-root`** — enable automated sync for that app only after **`noble.yml`** completes (**`clusters/noble/bootstrap/argocd/README.md`** §5). Put Helm **`Application`** migrations under **`clusters/noble/bootstrap/argocd/app-of-apps/`**. diff --git a/clusters/noble/apps/kustomization.yaml b/clusters/noble/apps/kustomization.yaml index 9f58d5a..d0d72de 100644 --- a/clusters/noble/apps/kustomization.yaml +++ b/clusters/noble/apps/kustomization.yaml @@ -5,6 +5,3 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - homepage/application.yaml - - eclipse-che/application-devworkspace.yaml - - eclipse-che/application-operator.yaml - - eclipse-che/application-checluster.yaml diff --git a/clusters/noble/bootstrap/argocd/README.md b/clusters/noble/bootstrap/argocd/README.md index aa6338f..0190815 100644 --- a/clusters/noble/bootstrap/argocd/README.md +++ b/clusters/noble/bootstrap/argocd/README.md @@ -50,21 +50,56 @@ helm upgrade --install argocd argo/argo-cd -n argocd --create-namespace \ Use **Settings → Repositories** in the UI, or `argocd repo add` / a `Secret` of type `repository`. -## 4. App-of-apps (optional GitOps only) +## 4. App-of-apps (GitOps) -Bootstrap **platform** workloads (CNI, ingress, cert-manager, Kyverno, observability, etc.) are installed by -**`ansible/playbooks/noble.yml`** from **`clusters/noble/bootstrap/`** — not by Argo. **`clusters/noble/apps/kustomization.yaml`** is empty by default. +**Ansible** (`ansible/playbooks/noble.yml`) performs the **initial** install: Helm releases and **`kubectl apply -k clusters/noble/bootstrap`**. **Argo** then tracks the same git paths for ongoing reconciliation. -1. Edit **`root-application.yaml`**: set **`repoURL`** and **`targetRevision`** to this repository. The **`resources-finalizer.argocd.argoproj.io/background`** finalizer uses Argo’s path-qualified form so **`kubectl apply`** does not warn about finalizer names. -2. When you want Argo to manage specific apps, add **`Application`** manifests under **`clusters/noble/apps/`** (see **`clusters/noble/apps/README.md`**). -3. Apply the root: +1. Edit **`root-application.yaml`** and **`bootstrap-root-application.yaml`**: set **`repoURL`** and **`targetRevision`**. The **`resources-finalizer.argocd.argoproj.io/background`** finalizer uses Argo’s path-qualified form so **`kubectl apply`** does not warn about finalizer names. +2. Optional add-on apps: add **`Application`** manifests under **`clusters/noble/apps/`** (see **`clusters/noble/apps/README.md`**). +3. **Bootstrap kustomize** (namespaces, datasource, leaf **`Application`**s under **`argocd/app-of-apps/`**, etc.): **`noble-bootstrap-root`** syncs **`clusters/noble/bootstrap`**. It is created with **manual** sync only so Argo does not apply changes while **`noble.yml`** is still running. + + **`ansible/playbooks/noble.yml`** (role **`noble_argocd`**) applies both roots when **`noble_argocd_apply_root_application`** / **`noble_argocd_apply_bootstrap_root_application`** are true in **`ansible/group_vars/all.yml`**. ```bash kubectl apply -f clusters/noble/bootstrap/argocd/root-application.yaml + kubectl apply -f clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml ``` -If you migrated from GitOps-managed **`noble-platform`** / **`noble-kyverno`**, delete stale **`Application`** objects on -the cluster (see **`clusters/noble/apps/README.md`**) then re-apply the root. +If you migrated from older GitOps **`Application`** names, delete stale **`Application`** objects on the cluster (see **`clusters/noble/apps/README.md`**) then re-apply the roots. + +## 5. After Ansible: enable automated sync for **noble-bootstrap-root** + +Do this only after **`ansible-playbook playbooks/noble.yml`** has finished successfully (including **`noble_platform`** `kubectl apply -k` and any Helm stages you rely on). Until then, leave **manual** sync so Argo does not fight the playbook. + +**Required steps** + +1. Confirm the cluster matches git for kustomize output (optional): `kubectl kustomize clusters/noble/bootstrap | kubectl diff -f -` or inspect resources in the UI. +2. Register the git repo in Argo if you have not already (**§3**). +3. **Refresh** the app so Argo compares **`clusters/noble/bootstrap`** to the cluster: Argo UI → **noble-bootstrap-root** → **Refresh**, or: + + ```bash + argocd app get noble-bootstrap-root --refresh + ``` + +4. **Enable automated sync** (prune + self-heal), preserving **`CreateNamespace`**, using any one of: + + **kubectl** + + ```bash + kubectl patch application noble-bootstrap-root -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true},"syncOptions":["CreateNamespace=true"]}}}' + ``` + + **argocd** CLI (logged in) + + ```bash + argocd app set noble-bootstrap-root --sync-policy automated --auto-prune --self-heal + ``` + + **UI:** open **noble-bootstrap-root** → **App Details** → enable **AUTO-SYNC** (and **Prune** / **Self Heal** if shown). + +5. Trigger a sync if the app does not go green immediately: **Sync** in the UI, or `argocd app sync noble-bootstrap-root`. + +After this, **git** is the source of truth for everything under **`clusters/noble/bootstrap/kustomization.yaml`** (including **`argocd/app-of-apps/`**). Helm-managed platform components remain whatever Ansible last installed until you model them as Argo **`Application`**s under **`app-of-apps/`** and stop installing them from Ansible. ## Versions diff --git a/clusters/noble/bootstrap/argocd/app-of-apps/kustomization.yaml b/clusters/noble/bootstrap/argocd/app-of-apps/kustomization.yaml new file mode 100644 index 0000000..602653c --- /dev/null +++ b/clusters/noble/bootstrap/argocd/app-of-apps/kustomization.yaml @@ -0,0 +1,6 @@ +# Sub-kustomization included by **clusters/noble/bootstrap/kustomization.yaml**. Leaf **Application** / +# **AppProject** resources (Helm apps you migrate off raw **helm upgrade** in Ansible). Synced with the +# rest of **clusters/noble/bootstrap** via **noble-bootstrap-root** once automated sync is enabled. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: [] diff --git a/clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml b/clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml new file mode 100644 index 0000000..f917d03 --- /dev/null +++ b/clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml @@ -0,0 +1,29 @@ +# **noble-bootstrap-root** — Kustomize app-of-apps for **clusters/noble/bootstrap** (same tree as +# **ansible/playbooks/noble.yml** → **noble_platform** `kubectl apply -k`). +# +# **Initial deploy:** Ansible is the only writer; **automated sync is off** so Argo does not reconcile +# during **noble.yml**. **After** the playbook finishes, enable automated sync (see **README.md** §5) +# so git becomes the source of truth for this kustomize output. +# +# Edit **spec.source.repoURL** / **targetRevision** for your remote. +# +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: noble-bootstrap-root + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io/background +spec: + project: default + source: + repoURL: https://gitea.pcenicni.ca/gsdavidp/home-server.git + targetRevision: HEAD + path: clusters/noble/bootstrap + destination: + server: https://kubernetes.default.svc + namespace: argocd + # Manual sync until you enable automation after Ansible (see README.md §5). + syncPolicy: + syncOptions: + - CreateNamespace=true diff --git a/clusters/noble/bootstrap/argocd/root-application.yaml b/clusters/noble/bootstrap/argocd/root-application.yaml index 7fd72e4..2f466f0 100644 --- a/clusters/noble/bootstrap/argocd/root-application.yaml +++ b/clusters/noble/bootstrap/argocd/root-application.yaml @@ -3,8 +3,10 @@ # 1. Set spec.source.repoURL (and targetRevision — **HEAD** tracks the remote default branch) to this repo. # 2. kubectl apply -f clusters/noble/bootstrap/argocd/root-application.yaml # -# **clusters/noble/apps** holds optional **Application** manifests. Core platform is installed by -# **ansible/playbooks/noble.yml** from **clusters/noble/bootstrap/**. +# **clusters/noble/apps** holds optional **Application** manifests. Core platform Helm + kustomize is +# installed by **ansible/playbooks/noble.yml** from **clusters/noble/bootstrap/**. **bootstrap-root-application.yaml** +# registers **noble-bootstrap-root** for the same kustomize tree (**manual** sync until you enable +# automation after the playbook — see **README.md** §5). # apiVersion: argoproj.io/v1alpha1 kind: Application diff --git a/clusters/noble/bootstrap/kustomization.yaml b/clusters/noble/bootstrap/kustomization.yaml index bebf821..88e7293 100644 --- a/clusters/noble/bootstrap/kustomization.yaml +++ b/clusters/noble/bootstrap/kustomization.yaml @@ -1,6 +1,8 @@ # Ansible bootstrap: plain Kustomize (namespaces + extra YAML). Helm installs are driven by # **ansible/playbooks/noble.yml** (role **noble_platform**) — avoids **kustomize --enable-helm** in-repo. -# Optional GitOps workloads live under **../apps/** (Argo **noble-root**). +# Optional GitOps: **../apps/** (Argo **noble-root**); leaf **Application**s under **argocd/app-of-apps/**. +# **noble-bootstrap-root** (Argo) uses this same path — enable automated sync only after **noble.yml** +# completes (see **argocd/README.md** §5). apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization @@ -14,3 +16,4 @@ resources: - velero/longhorn-volumesnapshotclass.yaml - headlamp/namespace.yaml - grafana-loki-datasource/loki-datasource.yaml + - argocd/app-of-apps diff --git a/docs/architecture.md b/docs/architecture.md index 59bb976..feaed55 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -214,7 +214,7 @@ See [`talos/CLUSTER-BUILD.md`](../talos/CLUSTER-BUILD.md) for the authoritative ## Narrative -The **noble** environment is a **Talos** lab cluster on **`192.168.50.0/24`** with **three control plane nodes and one worker**, schedulable workloads on control planes enabled, and the Kubernetes API exposed through **kube-vip** at **`192.168.50.230`**. **Cilium** provides the CNI after Talos bootstrap with **`cni: none`**; **MetalLB** advertises **`192.168.50.210`–`192.168.50.229`**, pinning **Argo CD** to **`192.168.50.210`** and **Traefik** to **`192.168.50.211`** for **`*.apps.noble.lab.pcenicni.dev`**. **cert-manager** issues certificates for Traefik Ingresses; **GitOps** is **Ansible-driven Helm** for the platform (**`clusters/noble/bootstrap/`**) plus optional **Argo CD** app-of-apps (**`clusters/noble/apps/`**, **`clusters/noble/bootstrap/argocd/`**). **Observability** uses **kube-prometheus-stack** in **`monitoring`**, **Loki** and **Fluent Bit** with Grafana wired via a **ConfigMap** datasource, with **Longhorn** PVCs for Prometheus, Grafana, Alertmanager, and Loki. **Secrets** in git use **SOPS** + **age** under **`clusters/noble/secrets/`**; **Kyverno** enforces **Pod Security Standards baseline** in **Audit**. **Public** access uses **Newt** to **Pangolin** with **CNAME** and Integration API steps as documented—not generic in-cluster public DNS. +The **noble** environment is a **Talos** lab cluster on **`192.168.50.0/24`** with **three control plane nodes and one worker**, schedulable workloads on control planes enabled, and the Kubernetes API exposed through **kube-vip** at **`192.168.50.230`**. **Cilium** provides the CNI after Talos bootstrap with **`cni: none`**; **MetalLB** advertises **`192.168.50.210`–`192.168.50.229`**, pinning **Argo CD** to **`192.168.50.210`** and **Traefik** to **`192.168.50.211`** for **`*.apps.noble.lab.pcenicni.dev`**. **cert-manager** issues certificates for Traefik Ingresses; **GitOps** is **Ansible** for the **initial** platform install (**`clusters/noble/bootstrap/`**), then **Argo CD** for the kustomize tree (**`noble-bootstrap-root`** → **`clusters/noble/bootstrap`**) and optional apps (**`noble-root`** → **`clusters/noble/apps/`**) once automated sync is enabled after **`noble.yml`** (see **`clusters/noble/bootstrap/argocd/README.md`** §5). **Observability** uses **kube-prometheus-stack** in **`monitoring`**, **Loki** and **Fluent Bit** with Grafana wired via a **ConfigMap** datasource, with **Longhorn** PVCs for Prometheus, Grafana, Alertmanager, and Loki. **Secrets** in git use **SOPS** + **age** under **`clusters/noble/secrets/`**; **Kyverno** enforces **Pod Security Standards baseline** in **Audit**. **Public** access uses **Newt** to **Pangolin** with **CNAME** and Integration API steps as documented—not generic in-cluster public DNS. --- diff --git a/talos/CLUSTER-BUILD.md b/talos/CLUSTER-BUILD.md index a8725bc..a7673fe 100644 --- a/talos/CLUSTER-BUILD.md +++ b/talos/CLUSTER-BUILD.md @@ -16,7 +16,7 @@ Lab stack is **up** on-cluster through **Phase D**–**F** and **Phase G** (**`t - **Traefik** Helm **39.0.6** / app **v3.6.11** — `clusters/noble/bootstrap/traefik/`; **`Service`** **`LoadBalancer`** **`EXTERNAL-IP` `192.168.50.211`**; **`IngressClass`** **`traefik`** (default). Point **`*.apps.noble.lab.pcenicni.dev`** at **`192.168.50.211`**. MetalLB pool verification was done before replacing the temporary nginx test with Traefik. - **cert-manager** Helm **v1.20.0** / app **v1.20.0** — `clusters/noble/bootstrap/cert-manager/`; **`ClusterIssuer`** **`letsencrypt-staging`** and **`letsencrypt-prod`** (**DNS-01** via **Cloudflare** for **`pcenicni.dev`**, Secret **`cloudflare-dns-api-token`** in **`cert-manager`**); ACME email **`certificates@noble.lab.pcenicni.dev`** (edit in manifests if you want a different mailbox). - **Newt** Helm **1.2.0** / app **1.10.1** — `clusters/noble/bootstrap/newt/` (**fossorial/newt**); Pangolin site tunnel — **`newt-pangolin-auth`** Secret (**`PANGOLIN_ENDPOINT`**, **`NEWT_ID`**, **`NEWT_SECRET`**). Store credentials in git with **SOPS** (`clusters/noble/secrets/newt-pangolin-auth.secret.yaml`, **`age-key.txt`**, **`.sops.yaml`**) — see **`clusters/noble/secrets/README.md`**. **Public DNS** is **not** automated with ExternalDNS: **CNAME** records at your DNS host per Pangolin’s domain instructions, plus **Integration API** for HTTP resources/targets — see **`clusters/noble/bootstrap/newt/README.md`**. LAN access to Traefik can still use **`*.apps.noble.lab.pcenicni.dev`** → **`192.168.50.211`** (split horizon / local resolver). -- **Argo CD** Helm **9.4.17** / app **v3.3.6** — `clusters/noble/bootstrap/argocd/`; **`argocd-server`** **`LoadBalancer`** **`192.168.50.210`**; app-of-apps root syncs **`clusters/noble/apps/`** (edit **`root-application.yaml`** `repoURL` before applying). +- **Argo CD** Helm **9.4.17** / app **v3.3.6** — `clusters/noble/bootstrap/argocd/`; **`argocd-server`** **`LoadBalancer`** **`192.168.50.210`**; **`noble-root`** → **`clusters/noble/apps/`**; **`noble-bootstrap-root`** → **`clusters/noble/bootstrap`** (manual sync until **`argocd/README.md`** §5 after **`noble.yml`**). Edit **`repoURL`** in both root **`Application`** files before applying. - **kube-prometheus-stack** — Helm chart **82.15.1** — `clusters/noble/bootstrap/kube-prometheus-stack/` (**namespace** `monitoring`, PSA **privileged** — **node-exporter** needs host mounts); **Longhorn** PVCs for Prometheus, Grafana, Alertmanager; **node-exporter** DaemonSet **4/4**. **Grafana Ingress:** **`https://grafana.apps.noble.lab.pcenicni.dev`** (Traefik **`ingressClassName: traefik`**, **`cert-manager.io/cluster-issuer: letsencrypt-prod`**). **Loki** datasource in Grafana: ConfigMap **`clusters/noble/bootstrap/grafana-loki-datasource/loki-datasource.yaml`** (sidecar label **`grafana_datasource: "1"`**) — not via **`grafana.additionalDataSources`** in the chart. **`helm upgrade --install` with `--wait` is silent until done** — use **`--timeout 30m`**; Grafana admin: Secret **`kube-prometheus-grafana`**, keys **`admin-user`** / **`admin-password`**. - **Loki** + **Fluent Bit** — **`grafana/loki` 6.55.0** SingleBinary + **filesystem** on **Longhorn** (`clusters/noble/bootstrap/loki/`); **`loki.auth_enabled: false`**; **`chunksCache.enabled: false`** (no memcached chunk cache). **`fluent/fluent-bit` 0.56.0** → **`loki-gateway.loki.svc:80`** (`clusters/noble/bootstrap/fluent-bit/`); **`logging`** PSA **privileged**. **Grafana Explore:** **`kubectl apply -f clusters/noble/bootstrap/grafana-loki-datasource/loki-datasource.yaml`** then **Explore → Loki** (e.g. `{job="fluent-bit"}`). - **SOPS** — cluster **`Secret`** manifests under **`clusters/noble/secrets/`** encrypted with **age** (see **`.sops.yaml`**, **`age-key.txt`** gitignored); **`noble.yml`** decrypt-applies when the private key is present. @@ -86,7 +86,7 @@ Lab stack is **up** on-cluster through **Phase D**–**F** and **Phase G** (**`t | Traefik (Helm values) | `clusters/noble/bootstrap/traefik/` — `values.yaml`, `namespace.yaml`, `README.md` | | cert-manager (Helm + ClusterIssuers) | `clusters/noble/bootstrap/cert-manager/` — `values.yaml`, `namespace.yaml`, `kustomization.yaml`, `README.md` | | Newt / Pangolin tunnel (Helm) | `clusters/noble/bootstrap/newt/` — `values.yaml`, `namespace.yaml`, `README.md` | -| Argo CD (Helm) + optional app-of-apps | `clusters/noble/bootstrap/argocd/` — `values.yaml`, `root-application.yaml`, `README.md`; optional **`Application`** tree in **`clusters/noble/apps/`** | +| Argo CD (Helm) + app-of-apps | `clusters/noble/bootstrap/argocd/` — `values.yaml`, `root-application.yaml`, `bootstrap-root-application.yaml`, `app-of-apps/`, `README.md`; **`noble-root`** syncs **`clusters/noble/apps/`**; **`noble-bootstrap-root`** syncs **`clusters/noble/bootstrap`** (enable automation after **`noble.yml`**) | | kube-prometheus-stack (Helm values) | `clusters/noble/bootstrap/kube-prometheus-stack/` — `values.yaml`, `namespace.yaml` | | Grafana Loki datasource (ConfigMap; no chart change) | `clusters/noble/bootstrap/grafana-loki-datasource/loki-datasource.yaml` | | Loki (Helm values) | `clusters/noble/bootstrap/loki/` — `values.yaml`, `namespace.yaml` |