Enhance Argo CD integration by adding support for a bootstrap root application. Update group_vars/all.yml and role defaults to include noble_argocd_apply_bootstrap_root_application. Modify tasks to apply the bootstrap application conditionally. Revise documentation to clarify the GitOps workflow and the relationship between the core platform and optional applications. Remove outdated references and streamline the README for better user guidance.

ansible/README.md

@@ -79,7 +79,7 @@ ansible-playbook playbooks/noble.yml --tags velero -e noble_velero_install=true
 
 ### Variables — `group_vars/all.yml` and role defaults
 
-- **`group_vars/all.yml`:** **`noble_newt_install`**, **`noble_velero_install`**, **`noble_cert_manager_require_cloudflare_secret`**, **`noble_argocd_apply_root_application`**, **`noble_k8s_api_server_override`**, **`noble_k8s_api_server_auto_fallback`**, **`noble_k8s_api_server_fallback`**, **`noble_skip_k8s_health_check`**
+- **`group_vars/all.yml`:** **`noble_newt_install`**, **`noble_velero_install`**, **`noble_cert_manager_require_cloudflare_secret`**, **`noble_argocd_apply_root_application`**, **`noble_argocd_apply_bootstrap_root_application`**, **`noble_k8s_api_server_override`**, **`noble_k8s_api_server_auto_fallback`**, **`noble_k8s_api_server_fallback`**, **`noble_skip_k8s_health_check`**
 - **`roles/noble_platform/defaults/main.yml`:** **`noble_apply_sops_secrets`**, **`noble_sops_age_key_file`** (SOPS secrets under **`clusters/noble/secrets/`**)
 
 ## Roles

ansible/group_vars/all.yml

@@ -24,3 +24,5 @@ noble_velero_install: false
 
 # Argo CD — apply app-of-apps root Application (clusters/noble/bootstrap/argocd/root-application.yaml). Set false to skip.
 noble_argocd_apply_root_application: true
+# Bootstrap kustomize in Argo (**noble-bootstrap-root** → **clusters/noble/bootstrap**). Applied with manual sync; enable automation after **noble.yml** (see **clusters/noble/bootstrap/argocd/README.md** §5).
+noble_argocd_apply_bootstrap_root_application: true

ansible/roles/noble_argocd/defaults/main.yml

@@ -2,3 +2,5 @@
 # When true, applies clusters/noble/bootstrap/argocd/root-application.yaml (app-of-apps).
 # Edit spec.source.repoURL in that file if your Git remote differs.
 noble_argocd_apply_root_application: false
+# When true, applies clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml (noble-bootstrap-root; manual sync until README §5).
+noble_argocd_apply_bootstrap_root_application: true

ansible/roles/noble_argocd/tasks/main.yml

@@ -32,3 +32,15 @@
     KUBECONFIG: "{{ noble_kubeconfig }}"
   when: noble_argocd_apply_root_application | default(false) | bool
   changed_when: true
+
+- name: Apply Argo CD bootstrap app-of-apps Application
+  ansible.builtin.command:
+    argv:
+      - kubectl
+      - apply
+      - -f
+      - "{{ noble_repo_root }}/clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml"
+  environment:
+    KUBECONFIG: "{{ noble_kubeconfig }}"
+  when: noble_argocd_apply_bootstrap_root_application | default(false) | bool
+  changed_when: true

ansible/roles/noble_post_deploy/tasks/main.yml

@@ -9,6 +9,7 @@
 - name: Argo CD optional root Application (empty app-of-apps)
   ansible.builtin.debug:
     msg: >-
-      App-of-apps: noble.yml applies root-application.yaml when noble_argocd_apply_root_application is true
-      (group_vars/all.yml). Otherwise: kubectl apply -f clusters/noble/bootstrap/argocd/root-application.yaml
-      after editing spec.source.repoURL. Core platform is Ansible — see clusters/noble/apps/README.md
+      App-of-apps: noble.yml applies root-application.yaml when noble_argocd_apply_root_application is true;
+      bootstrap-root-application.yaml when noble_argocd_apply_bootstrap_root_application is true (group_vars/all.yml).
+      noble-bootstrap-root uses manual sync until you enable automation after the playbook —
+      clusters/noble/bootstrap/argocd/README.md §5. See clusters/noble/apps/README.md and that README.

clusters/noble/apps/README.md

@@ -2,6 +2,6 @@
 
 **Base cluster configuration** (CNI, MetalLB, ingress, cert-manager, storage, observability stack, policy, SOPS secrets path, etc.) is installed by **`ansible/playbooks/noble.yml`** from **`clusters/noble/bootstrap/`** — not from here.
 
-**`noble-root`** (`clusters/noble/bootstrap/argocd/root-application.yaml`) points at **`clusters/noble/apps`**. Add **`Application`** manifests (and optional **`AppProject`** definitions) under this directory only for workloads that are additive and do not subsume the Ansible-managed platform.
+**`noble-root`** (`clusters/noble/bootstrap/argocd/root-application.yaml`) points at **`clusters/noble/apps`**. Add **`Application`** manifests (and optional **`AppProject`** definitions) under this directory only for workloads that are additive and do not subsume the core platform.
 
-For an app-of-apps pattern, use a second-level **`Application`** that syncs a subdirectory (for example **`optional/`**) containing leaf **`Application`** resources.
+Bootstrap kustomize (namespaces, static YAML, leaf **`Application`**s) lives in **`clusters/noble/bootstrap/`** and is tracked by **`noble-bootstrap-root`** — enable automated sync for that app only after **`noble.yml`** completes (**`clusters/noble/bootstrap/argocd/README.md`** §5). Put Helm **`Application`** migrations under **`clusters/noble/bootstrap/argocd/app-of-apps/`**.

clusters/noble/apps/kustomization.yaml

@@ -5,6 +5,3 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - homepage/application.yaml
-  - eclipse-che/application-devworkspace.yaml
-  - eclipse-che/application-operator.yaml
-  - eclipse-che/application-checluster.yaml

clusters/noble/bootstrap/argocd/README.md

@@ -50,21 +50,56 @@ helm upgrade --install argocd argo/argo-cd -n argocd --create-namespace \
 
 Use **Settings → Repositories** in the UI, or `argocd repo add` / a `Secret` of type `repository`.
 
-## 4. App-of-apps (optional GitOps only)
+## 4. App-of-apps (GitOps)
 
-Bootstrap **platform** workloads (CNI, ingress, cert-manager, Kyverno, observability, etc.) are installed by
-**`ansible/playbooks/noble.yml`** from **`clusters/noble/bootstrap/`** — not by Argo. **`clusters/noble/apps/kustomization.yaml`** is empty by default.
+**Ansible** (`ansible/playbooks/noble.yml`) performs the **initial** install: Helm releases and **`kubectl apply -k clusters/noble/bootstrap`**. **Argo** then tracks the same git paths for ongoing reconciliation.
 
-1. Edit **`root-application.yaml`**: set **`repoURL`** and **`targetRevision`** to this repository. The **`resources-finalizer.argocd.argoproj.io/background`** finalizer uses Argo’s path-qualified form so **`kubectl apply`** does not warn about finalizer names.
-2. When you want Argo to manage specific apps, add **`Application`** manifests under **`clusters/noble/apps/`** (see **`clusters/noble/apps/README.md`**).
-3. Apply the root:
+1. Edit **`root-application.yaml`** and **`bootstrap-root-application.yaml`**: set **`repoURL`** and **`targetRevision`**. The **`resources-finalizer.argocd.argoproj.io/background`** finalizer uses Argo’s path-qualified form so **`kubectl apply`** does not warn about finalizer names.
+2. Optional add-on apps: add **`Application`** manifests under **`clusters/noble/apps/`** (see **`clusters/noble/apps/README.md`**).
+3. **Bootstrap kustomize** (namespaces, datasource, leaf **`Application`**s under **`argocd/app-of-apps/`**, etc.): **`noble-bootstrap-root`** syncs **`clusters/noble/bootstrap`**. It is created with **manual** sync only so Argo does not apply changes while **`noble.yml`** is still running.
+
+   **`ansible/playbooks/noble.yml`** (role **`noble_argocd`**) applies both roots when **`noble_argocd_apply_root_application`** / **`noble_argocd_apply_bootstrap_root_application`** are true in **`ansible/group_vars/all.yml`**.
 
    ```bash
    kubectl apply -f clusters/noble/bootstrap/argocd/root-application.yaml
+   kubectl apply -f clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml
    ```
 
-If you migrated from GitOps-managed **`noble-platform`** / **`noble-kyverno`**, delete stale **`Application`** objects on
-the cluster (see **`clusters/noble/apps/README.md`**) then re-apply the root.
+If you migrated from older GitOps **`Application`** names, delete stale **`Application`** objects on the cluster (see **`clusters/noble/apps/README.md`**) then re-apply the roots.
+
+## 5. After Ansible: enable automated sync for **noble-bootstrap-root**
+
+Do this only after **`ansible-playbook playbooks/noble.yml`** has finished successfully (including **`noble_platform`** `kubectl apply -k` and any Helm stages you rely on). Until then, leave **manual** sync so Argo does not fight the playbook.
+
+**Required steps**
+
+1. Confirm the cluster matches git for kustomize output (optional): `kubectl kustomize clusters/noble/bootstrap | kubectl diff -f -` or inspect resources in the UI.
+2. Register the git repo in Argo if you have not already (**§3**).
+3. **Refresh** the app so Argo compares **`clusters/noble/bootstrap`** to the cluster: Argo UI → **noble-bootstrap-root** → **Refresh**, or:
+
+   ```bash
+   argocd app get noble-bootstrap-root --refresh
+   ```
+
+4. **Enable automated sync** (prune + self-heal), preserving **`CreateNamespace`**, using any one of:
+
+   **kubectl**
+
+   ```bash
+   kubectl patch application noble-bootstrap-root -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true},"syncOptions":["CreateNamespace=true"]}}}'
+   ```
+
+   **argocd** CLI (logged in)
+
+   ```bash
+   argocd app set noble-bootstrap-root --sync-policy automated --auto-prune --self-heal
+   ```
+
+   **UI:** open **noble-bootstrap-root** → **App Details** → enable **AUTO-SYNC** (and **Prune** / **Self Heal** if shown).
+
+5. Trigger a sync if the app does not go green immediately: **Sync** in the UI, or `argocd app sync noble-bootstrap-root`.
+
+After this, **git** is the source of truth for everything under **`clusters/noble/bootstrap/kustomization.yaml`** (including **`argocd/app-of-apps/`**). Helm-managed platform components remain whatever Ansible last installed until you model them as Argo **`Application`**s under **`app-of-apps/`** and stop installing them from Ansible.
 
 ## Versions
 

clusters/noble/bootstrap/argocd/app-of-apps/kustomization.yaml

@@ -0,0 +1,6 @@
+# Sub-kustomization included by **clusters/noble/bootstrap/kustomization.yaml**. Leaf **Application** /
+# **AppProject** resources (Helm apps you migrate off raw **helm upgrade** in Ansible). Synced with the
+# rest of **clusters/noble/bootstrap** via **noble-bootstrap-root** once automated sync is enabled.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources: []

clusters/noble/bootstrap/argocd/bootstrap-root-application.yaml

@@ -0,0 +1,29 @@
+# **noble-bootstrap-root** — Kustomize app-of-apps for **clusters/noble/bootstrap** (same tree as
+# **ansible/playbooks/noble.yml** → **noble_platform** `kubectl apply -k`).
+#
+# **Initial deploy:** Ansible is the only writer; **automated sync is off** so Argo does not reconcile
+# during **noble.yml**. **After** the playbook finishes, enable automated sync (see **README.md** §5)
+# so git becomes the source of truth for this kustomize output.
+#
+# Edit **spec.source.repoURL** / **targetRevision** for your remote.
+#
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: noble-bootstrap-root
+  namespace: argocd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io/background
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.pcenicni.ca/gsdavidp/home-server.git
+    targetRevision: HEAD
+    path: clusters/noble/bootstrap
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: argocd
+  # Manual sync until you enable automation after Ansible (see README.md §5).
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

clusters/noble/bootstrap/argocd/root-application.yaml

@@ -3,8 +3,10 @@
 # 1. Set spec.source.repoURL (and targetRevision — **HEAD** tracks the remote default branch) to this repo.
 # 2. kubectl apply -f clusters/noble/bootstrap/argocd/root-application.yaml
 #
-# **clusters/noble/apps** holds optional **Application** manifests. Core platform is installed by
-# **ansible/playbooks/noble.yml** from **clusters/noble/bootstrap/**.
+# **clusters/noble/apps** holds optional **Application** manifests. Core platform Helm + kustomize is
+# installed by **ansible/playbooks/noble.yml** from **clusters/noble/bootstrap/**. **bootstrap-root-application.yaml**
+# registers **noble-bootstrap-root** for the same kustomize tree (**manual** sync until you enable
+# automation after the playbook — see **README.md** §5).
 #
 apiVersion: argoproj.io/v1alpha1
 kind: Application

clusters/noble/bootstrap/kustomization.yaml

@@ -1,6 +1,8 @@
 # Ansible bootstrap: plain Kustomize (namespaces + extra YAML). Helm installs are driven by
 # **ansible/playbooks/noble.yml** (role **noble_platform**) — avoids **kustomize --enable-helm** in-repo.
-# Optional GitOps workloads live under **../apps/** (Argo **noble-root**).
+# Optional GitOps: **../apps/** (Argo **noble-root**); leaf **Application**s under **argocd/app-of-apps/**.
+# **noble-bootstrap-root** (Argo) uses this same path — enable automated sync only after **noble.yml**
+# completes (see **argocd/README.md** §5).
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 
@@ -14,3 +16,4 @@ resources:
   - velero/longhorn-volumesnapshotclass.yaml
   - headlamp/namespace.yaml
   - grafana-loki-datasource/loki-datasource.yaml
+  - argocd/app-of-apps

docs/architecture.md

@@ -214,7 +214,7 @@ See [`talos/CLUSTER-BUILD.md`](../talos/CLUSTER-BUILD.md) for the authoritative
 
 ## Narrative
 
-The **noble** environment is a **Talos** lab cluster on **`192.168.50.0/24`** with **three control plane nodes and one worker**, schedulable workloads on control planes enabled, and the Kubernetes API exposed through **kube-vip** at **`192.168.50.230`**. **Cilium** provides the CNI after Talos bootstrap with **`cni: none`**; **MetalLB** advertises **`192.168.50.210`–`192.168.50.229`**, pinning **Argo CD** to **`192.168.50.210`** and **Traefik** to **`192.168.50.211`** for **`*.apps.noble.lab.pcenicni.dev`**. **cert-manager** issues certificates for Traefik Ingresses; **GitOps** is **Ansible-driven Helm** for the platform (**`clusters/noble/bootstrap/`**) plus optional **Argo CD** app-of-apps (**`clusters/noble/apps/`**, **`clusters/noble/bootstrap/argocd/`**). **Observability** uses **kube-prometheus-stack** in **`monitoring`**, **Loki** and **Fluent Bit** with Grafana wired via a **ConfigMap** datasource, with **Longhorn** PVCs for Prometheus, Grafana, Alertmanager, and Loki. **Secrets** in git use **SOPS** + **age** under **`clusters/noble/secrets/`**; **Kyverno** enforces **Pod Security Standards baseline** in **Audit**. **Public** access uses **Newt** to **Pangolin** with **CNAME** and Integration API steps as documented—not generic in-cluster public DNS.
+The **noble** environment is a **Talos** lab cluster on **`192.168.50.0/24`** with **three control plane nodes and one worker**, schedulable workloads on control planes enabled, and the Kubernetes API exposed through **kube-vip** at **`192.168.50.230`**. **Cilium** provides the CNI after Talos bootstrap with **`cni: none`**; **MetalLB** advertises **`192.168.50.210`–`192.168.50.229`**, pinning **Argo CD** to **`192.168.50.210`** and **Traefik** to **`192.168.50.211`** for **`*.apps.noble.lab.pcenicni.dev`**. **cert-manager** issues certificates for Traefik Ingresses; **GitOps** is **Ansible** for the **initial** platform install (**`clusters/noble/bootstrap/`**), then **Argo CD** for the kustomize tree (**`noble-bootstrap-root`** → **`clusters/noble/bootstrap`**) and optional apps (**`noble-root`** → **`clusters/noble/apps/`**) once automated sync is enabled after **`noble.yml`** (see **`clusters/noble/bootstrap/argocd/README.md`** §5). **Observability** uses **kube-prometheus-stack** in **`monitoring`**, **Loki** and **Fluent Bit** with Grafana wired via a **ConfigMap** datasource, with **Longhorn** PVCs for Prometheus, Grafana, Alertmanager, and Loki. **Secrets** in git use **SOPS** + **age** under **`clusters/noble/secrets/`**; **Kyverno** enforces **Pod Security Standards baseline** in **Audit**. **Public** access uses **Newt** to **Pangolin** with **CNAME** and Integration API steps as documented—not generic in-cluster public DNS.
 
 ---
 

talos/CLUSTER-BUILD.md

@@ -16,7 +16,7 @@ Lab stack is **up** on-cluster through **Phase D**–**F** and **Phase G** (**`t
 - **Traefik** Helm **39.0.6** / app **v3.6.11** — `clusters/noble/bootstrap/traefik/`; **`Service`** **`LoadBalancer`** **`EXTERNAL-IP` `192.168.50.211`**; **`IngressClass`** **`traefik`** (default). Point **`*.apps.noble.lab.pcenicni.dev`** at **`192.168.50.211`**. MetalLB pool verification was done before replacing the temporary nginx test with Traefik.
 - **cert-manager** Helm **v1.20.0** / app **v1.20.0** — `clusters/noble/bootstrap/cert-manager/`; **`ClusterIssuer`** **`letsencrypt-staging`** and **`letsencrypt-prod`** (**DNS-01** via **Cloudflare** for **`pcenicni.dev`**, Secret **`cloudflare-dns-api-token`** in **`cert-manager`**); ACME email **`certificates@noble.lab.pcenicni.dev`** (edit in manifests if you want a different mailbox).
 - **Newt** Helm **1.2.0** / app **1.10.1** — `clusters/noble/bootstrap/newt/` (**fossorial/newt**); Pangolin site tunnel — **`newt-pangolin-auth`** Secret (**`PANGOLIN_ENDPOINT`**, **`NEWT_ID`**, **`NEWT_SECRET`**). Store credentials in git with **SOPS** (`clusters/noble/secrets/newt-pangolin-auth.secret.yaml`, **`age-key.txt`**, **`.sops.yaml`**) — see **`clusters/noble/secrets/README.md`**. **Public DNS** is **not** automated with ExternalDNS: **CNAME** records at your DNS host per Pangolin’s domain instructions, plus **Integration API** for HTTP resources/targets — see **`clusters/noble/bootstrap/newt/README.md`**. LAN access to Traefik can still use **`*.apps.noble.lab.pcenicni.dev`** → **`192.168.50.211`** (split horizon / local resolver).
-- **Argo CD** Helm **9.4.17** / app **v3.3.6** — `clusters/noble/bootstrap/argocd/`; **`argocd-server`** **`LoadBalancer`** **`192.168.50.210`**; app-of-apps root syncs **`clusters/noble/apps/`** (edit **`root-application.yaml`** `repoURL` before applying).
+- **Argo CD** Helm **9.4.17** / app **v3.3.6** — `clusters/noble/bootstrap/argocd/`; **`argocd-server`** **`LoadBalancer`** **`192.168.50.210`**; **`noble-root`** → **`clusters/noble/apps/`**; **`noble-bootstrap-root`** → **`clusters/noble/bootstrap`** (manual sync until **`argocd/README.md`** §5 after **`noble.yml`**). Edit **`repoURL`** in both root **`Application`** files before applying.
 - **kube-prometheus-stack** — Helm chart **82.15.1** — `clusters/noble/bootstrap/kube-prometheus-stack/` (**namespace** `monitoring`, PSA **privileged** — **node-exporter** needs host mounts); **Longhorn** PVCs for Prometheus, Grafana, Alertmanager; **node-exporter** DaemonSet **4/4**. **Grafana Ingress:** **`https://grafana.apps.noble.lab.pcenicni.dev`** (Traefik **`ingressClassName: traefik`**, **`cert-manager.io/cluster-issuer: letsencrypt-prod`**). **Loki** datasource in Grafana: ConfigMap **`clusters/noble/bootstrap/grafana-loki-datasource/loki-datasource.yaml`** (sidecar label **`grafana_datasource: "1"`**) — not via **`grafana.additionalDataSources`** in the chart. **`helm upgrade --install` with `--wait` is silent until done** — use **`--timeout 30m`**; Grafana admin: Secret **`kube-prometheus-grafana`**, keys **`admin-user`** / **`admin-password`**.
 - **Loki** + **Fluent Bit** — **`grafana/loki` 6.55.0** SingleBinary + **filesystem** on **Longhorn** (`clusters/noble/bootstrap/loki/`); **`loki.auth_enabled: false`**; **`chunksCache.enabled: false`** (no memcached chunk cache). **`fluent/fluent-bit` 0.56.0** → **`loki-gateway.loki.svc:80`** (`clusters/noble/bootstrap/fluent-bit/`); **`logging`** PSA **privileged**. **Grafana Explore:** **`kubectl apply -f clusters/noble/bootstrap/grafana-loki-datasource/loki-datasource.yaml`** then **Explore → Loki** (e.g. `{job="fluent-bit"}`).
 - **SOPS** — cluster **`Secret`** manifests under **`clusters/noble/secrets/`** encrypted with **age** (see **`.sops.yaml`**, **`age-key.txt`** gitignored); **`noble.yml`** decrypt-applies when the private key is present.
@@ -86,7 +86,7 @@ Lab stack is **up** on-cluster through **Phase D**–**F** and **Phase G** (**`t
 | Traefik (Helm values) | `clusters/noble/bootstrap/traefik/` — `values.yaml`, `namespace.yaml`, `README.md` |
 | cert-manager (Helm + ClusterIssuers) | `clusters/noble/bootstrap/cert-manager/` — `values.yaml`, `namespace.yaml`, `kustomization.yaml`, `README.md` |
 | Newt / Pangolin tunnel (Helm) | `clusters/noble/bootstrap/newt/` — `values.yaml`, `namespace.yaml`, `README.md` |
-| Argo CD (Helm) + optional app-of-apps | `clusters/noble/bootstrap/argocd/` — `values.yaml`, `root-application.yaml`, `README.md`; optional **`Application`** tree in **`clusters/noble/apps/`** |
+| Argo CD (Helm) + app-of-apps | `clusters/noble/bootstrap/argocd/` — `values.yaml`, `root-application.yaml`, `bootstrap-root-application.yaml`, `app-of-apps/`, `README.md`; **`noble-root`** syncs **`clusters/noble/apps/`**; **`noble-bootstrap-root`** syncs **`clusters/noble/bootstrap`** (enable automation after **`noble.yml`**) |
 | kube-prometheus-stack (Helm values) | `clusters/noble/bootstrap/kube-prometheus-stack/` — `values.yaml`, `namespace.yaml` |
 | Grafana Loki datasource (ConfigMap; no chart change) | `clusters/noble/bootstrap/grafana-loki-datasource/loki-datasource.yaml` |
 | Loki (Helm values) | `clusters/noble/bootstrap/loki/` — `values.yaml`, `namespace.yaml` |