Refactor noble cluster configurations by removing deprecated Argo CD application management files and transitioning to a streamlined Ansible-driven installation approach. Update kustomization.yaml files to reflect the new structure, ensuring clarity on resource management. Introduce new namespaces and configurations for cert-manager, external-secrets, and logging components, enhancing the overall deployment process. Add detailed README.md documentation for each component to guide users through the setup and management of the noble lab environment.

clusters/noble/bootstrap/vault/README.md

@@ -0,0 +1,162 @@
+# HashiCorp Vault (noble)
+
+Standalone Vault with **file** storage on a **Longhorn** PVC (`server.dataStorage`). The listener uses **HTTP** (`global.tlsDisable: true`) for in-cluster use; add TLS at the listener when exposing outside the cluster.
+
+- **Chart:** `hashicorp/vault` **0.32.0** (Vault **1.21.2**)
+- **Namespace:** `vault`
+
+## Install
+
+```bash
+helm repo add hashicorp https://helm.releases.hashicorp.com
+helm repo update
+kubectl apply -f clusters/noble/apps/vault/namespace.yaml
+helm upgrade --install vault hashicorp/vault -n vault \
+  --version 0.32.0 -f clusters/noble/apps/vault/values.yaml --wait --timeout 15m
+```
+
+Verify:
+
+```bash
+kubectl -n vault get pods,pvc,svc
+kubectl -n vault exec -i sts/vault -- vault status
+```
+
+## Cilium network policy (Phase G)
+
+After **Cilium** is up, optionally restrict HTTP access to the Vault server pods (**TCP 8200**) to **`external-secrets`** and same-namespace clients:
+
+```bash
+kubectl apply -f clusters/noble/apps/vault/cilium-network-policy.yaml
+```
+
+If you add workloads in other namespaces that call Vault, extend **`ingress`** in that manifest.
+
+## Initialize and unseal (first time)
+
+From a workstation with `kubectl` (or `kubectl exec` into any pod with `vault` CLI):
+
+```bash
+kubectl -n vault exec -i sts/vault -- vault operator init -key-shares=1 -key-threshold=1
+```
+
+**Lab-only:** `-key-shares=1 -key-threshold=1` keeps a single unseal key. For stronger Shamir splits, use more shares and store them safely.
+
+Save the **Unseal Key** and **Root Token** offline. Then unseal once:
+
+```bash
+kubectl -n vault exec -i sts/vault -- vault operator unseal
+# paste unseal key
+```
+
+Or create the Secret used by the optional CronJob and apply it:
+
+```bash
+kubectl -n vault create secret generic vault-unseal-key --from-literal=key='YOUR_UNSEAL_KEY'
+kubectl apply -f clusters/noble/apps/vault/unseal-cronjob.yaml
+```
+
+The CronJob runs every minute and unseals if Vault is sealed and the Secret is present.
+
+## Auto-unseal note
+
+Vault **OSS** auto-unseal uses cloud KMS (AWS, GCP, Azure, OCI), **Transit** (another Vault), etc. There is no first-class “Kubernetes Secret” seal. This repo uses an optional **CronJob** as a **lab** substitute. Production clusters should use a supported seal backend.
+
+## Kubernetes auth (External Secrets / ClusterSecretStore)
+
+**One-shot:** from the repo root, `export KUBECONFIG=talos/kubeconfig` and `export VAULT_TOKEN=…`, then run **`./clusters/noble/apps/vault/configure-kubernetes-auth.sh`** (idempotent). Then **`kubectl apply -f clusters/noble/apps/external-secrets/examples/vault-cluster-secret-store.yaml`** on its own line (shell comments **`# …`** on the same line are parsed as extra `kubectl` args and break `apply`). **`kubectl get clustersecretstore vault`** should show **READY=True** after a few seconds.
+
+Run these **from your workstation** (needs `kubectl`; no local `vault` binary required). Use a **short-lived admin token** or the root token **only in your shell** — do not paste tokens into logs or chat.
+
+**1. Enable the auth method** (skip if already done):
+
+```bash
+kubectl -n vault exec -it sts/vault -- sh -c '
+  export VAULT_ADDR=http://127.0.0.1:8200
+  export VAULT_TOKEN="YOUR_ROOT_OR_ADMIN_TOKEN"
+  vault auth enable kubernetes
+'
+```
+
+**2. Configure `auth/kubernetes`** — the API **issuer** must match the `iss` claim on service account JWTs. With **kube-vip** / a custom API URL, discover it from the cluster (do not assume `kubernetes.default`):
+
+```bash
+ISSUER=$(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer)
+REVIEWER=$(kubectl -n vault create token vault --duration=8760h)
+CA_B64=$(kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')
+```
+
+Then apply config **inside** the Vault pod (environment variables are passed in with `env` so quoting stays correct):
+
+```bash
+export VAULT_TOKEN="YOUR_ROOT_OR_ADMIN_TOKEN"
+export ISSUER REVIEWER CA_B64
+kubectl -n vault exec -i sts/vault -- env \
+  VAULT_ADDR=http://127.0.0.1:8200 \
+  VAULT_TOKEN="$VAULT_TOKEN" \
+  CA_B64="$CA_B64" \
+  REVIEWER="$REVIEWER" \
+  ISSUER="$ISSUER" \
+  sh -ec '
+  echo "$CA_B64" | base64 -d > /tmp/k8s-ca.crt
+  vault write auth/kubernetes/config \
+    kubernetes_host="https://kubernetes.default.svc:443" \
+    kubernetes_ca_cert=@/tmp/k8s-ca.crt \
+    token_reviewer_jwt="$REVIEWER" \
+    issuer="$ISSUER"
+'
+```
+
+**3. KV v2** at path `secret` (skip if already enabled):
+
+```bash
+kubectl -n vault exec -it sts/vault -- sh -c '
+  export VAULT_ADDR=http://127.0.0.1:8200
+  export VAULT_TOKEN="YOUR_ROOT_OR_ADMIN_TOKEN"
+  vault secrets enable -path=secret kv-v2
+'
+```
+
+**4. Policy + role** for the External Secrets operator SA (`external-secrets` / `external-secrets`):
+
+```bash
+kubectl -n vault exec -it sts/vault -- sh -c '
+  export VAULT_ADDR=http://127.0.0.1:8200
+  export VAULT_TOKEN="YOUR_ROOT_OR_ADMIN_TOKEN"
+  vault policy write external-secrets - <<EOF
+path "secret/data/*" {
+  capabilities = ["read", "list"]
+}
+path "secret/metadata/*" {
+  capabilities = ["read", "list"]
+}
+EOF
+  vault write auth/kubernetes/role/external-secrets \
+    bound_service_account_names=external-secrets \
+    bound_service_account_namespaces=external-secrets \
+    policies=external-secrets \
+    ttl=24h
+'
+```
+
+**5. Apply** **`clusters/noble/apps/external-secrets/examples/vault-cluster-secret-store.yaml`** if you have not already, then verify:
+
+```bash
+kubectl describe clustersecretstore vault
+```
+
+See also [Kubernetes auth](https://developer.hashicorp.com/vault/docs/auth/kubernetes#configuration).
+
+## TLS and External Secrets
+
+`values.yaml` disables TLS on the Vault listener. The **`ClusterSecretStore`** example uses **`http://vault.vault.svc.cluster.local:8200`**. If you enable TLS on the listener, switch the URL to **`https://`** and configure **`caBundle`** or **`caProvider`** on the store.
+
+## UI
+
+Port-forward:
+
+```bash
+kubectl -n vault port-forward svc/vault-ui 8200:8200
+```
+
+Open `http://127.0.0.1:8200` and log in with the root token (rotate for production workflows).

clusters/noble/bootstrap/vault/cilium-network-policy.yaml

@@ -0,0 +1,40 @@
+# CiliumNetworkPolicy — restrict who may reach Vault HTTP listener (8200).
+# Apply after Cilium is healthy: kubectl apply -f clusters/noble/apps/vault/cilium-network-policy.yaml
+#
+# Ingress-only policy: egress from Vault is unchanged (Kubernetes auth needs API + DNS).
+# Extend ingress rules if other namespaces must call Vault (e.g. app workloads).
+#
+# Ref: https://docs.cilium.io/en/stable/security/policy/language/
+---
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: vault-http-ingress
+  namespace: vault
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: vault
+      component: server
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": external-secrets
+      toPorts:
+        - ports:
+            - port: "8200"
+              protocol: TCP
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": traefik
+      toPorts:
+        - ports:
+            - port: "8200"
+              protocol: TCP
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": vault
+      toPorts:
+        - ports:
+            - port: "8200"
+              protocol: TCP

clusters/noble/bootstrap/vault/configure-kubernetes-auth.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# Configure Vault Kubernetes auth + KV v2 + policy/role for External Secrets Operator.
+# Requires: kubectl (cluster access), jq optional (openid issuer); Vault reachable via sts/vault.
+#
+# Usage (from repo root):
+#   export KUBECONFIG=talos/kubeconfig   # or your path
+#   export VAULT_TOKEN='…'               # root or admin token — never commit
+#   ./clusters/noble/apps/vault/configure-kubernetes-auth.sh
+#
+# Then: kubectl apply -f clusters/noble/apps/external-secrets/examples/vault-cluster-secret-store.yaml
+# Verify: kubectl describe clustersecretstore vault
+
+set -euo pipefail
+
+: "${VAULT_TOKEN:?Set VAULT_TOKEN to your Vault root or admin token}"
+
+ISSUER=$(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer)
+REVIEWER=$(kubectl -n vault create token vault --duration=8760h)
+CA_B64=$(kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')
+
+kubectl -n vault exec -i sts/vault -- env \
+  VAULT_ADDR=http://127.0.0.1:8200 \
+  VAULT_TOKEN="$VAULT_TOKEN" \
+  sh -ec '
+    set -e
+    vault auth list >/tmp/vauth.txt
+    grep -q "^kubernetes/" /tmp/vauth.txt || vault auth enable kubernetes
+  '
+
+kubectl -n vault exec -i sts/vault -- env \
+  VAULT_ADDR=http://127.0.0.1:8200 \
+  VAULT_TOKEN="$VAULT_TOKEN" \
+  CA_B64="$CA_B64" \
+  REVIEWER="$REVIEWER" \
+  ISSUER="$ISSUER" \
+  sh -ec '
+    echo "$CA_B64" | base64 -d > /tmp/k8s-ca.crt
+    vault write auth/kubernetes/config \
+      kubernetes_host="https://kubernetes.default.svc:443" \
+      kubernetes_ca_cert=@/tmp/k8s-ca.crt \
+      token_reviewer_jwt="$REVIEWER" \
+      issuer="$ISSUER"
+  '
+
+kubectl -n vault exec -i sts/vault -- env \
+  VAULT_ADDR=http://127.0.0.1:8200 \
+  VAULT_TOKEN="$VAULT_TOKEN" \
+  sh -ec '
+    set -e
+    vault secrets list >/tmp/vsec.txt
+    grep -q "^secret/" /tmp/vsec.txt || vault secrets enable -path=secret kv-v2
+  '
+
+kubectl -n vault exec -i sts/vault -- env \
+  VAULT_ADDR=http://127.0.0.1:8200 \
+  VAULT_TOKEN="$VAULT_TOKEN" \
+  sh -ec '
+    vault policy write external-secrets - <<EOF
+path "secret/data/*" {
+  capabilities = ["read", "list"]
+}
+path "secret/metadata/*" {
+  capabilities = ["read", "list"]
+}
+EOF
+    vault write auth/kubernetes/role/external-secrets \
+      bound_service_account_names=external-secrets \
+      bound_service_account_namespaces=external-secrets \
+      policies=external-secrets \
+      ttl=24h
+  '
+
+echo "Done. Issuer used: $ISSUER"
+echo ""
+echo "Next (each command on its own line — do not paste # comments after kubectl):"
+echo "  kubectl apply -f clusters/noble/apps/external-secrets/examples/vault-cluster-secret-store.yaml"
+echo "  kubectl get clustersecretstore vault"

clusters/noble/bootstrap/vault/namespace.yaml

@@ -0,0 +1,5 @@
+# HashiCorp Vault — apply before Helm.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: vault

clusters/noble/bootstrap/vault/unseal-cronjob.yaml

@@ -0,0 +1,63 @@
+# Optional lab auto-unseal: applies after Vault is initialized and Secret `vault-unseal-key` exists.
+#
+# 1) vault operator init -key-shares=1 -key-threshold=1  (lab only — single key)
+# 2) kubectl -n vault create secret generic vault-unseal-key --from-literal=key='YOUR_UNSEAL_KEY'
+# 3) kubectl apply -f clusters/noble/apps/vault/unseal-cronjob.yaml
+#
+# OSS Vault has no Kubernetes/KMS seal; this CronJob runs vault operator unseal when the server is sealed.
+# Protect the Secret with RBAC; prefer cloud KMS auto-unseal for real environments.
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: vault-auto-unseal
+  namespace: vault
+spec:
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 3
+  schedule: "*/1 * * * *"
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          restartPolicy: OnFailure
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 100
+            runAsGroup: 1000
+            seccompProfile:
+              type: RuntimeDefault
+          containers:
+            - name: unseal
+              image: hashicorp/vault:1.21.2
+              imagePullPolicy: IfNotPresent
+              securityContext:
+                allowPrivilegeEscalation: false
+                capabilities:
+                  drop:
+                    - ALL
+              env:
+                - name: VAULT_ADDR
+                  value: http://vault.vault.svc:8200
+              command:
+                - /bin/sh
+                - -ec
+                - |
+                  test -f /secrets/key || exit 0
+                  status="$(vault status -format=json 2>/dev/null || true)"
+                  echo "$status" | grep -q '"initialized":true' || exit 0
+                  echo "$status" | grep -q '"sealed":false' && exit 0
+                  vault operator unseal "$(cat /secrets/key)"
+              volumeMounts:
+                - name: unseal
+                  mountPath: /secrets
+                  readOnly: true
+          volumes:
+            - name: unseal
+              secret:
+                secretName: vault-unseal-key
+                optional: true
+                items:
+                  - key: key
+                    path: key

clusters/noble/bootstrap/vault/values.yaml

@@ -0,0 +1,62 @@
+# HashiCorp Vault — noble (standalone, file storage on Longhorn; TLS disabled on listener for in-cluster HTTP).
+#
+# helm repo add hashicorp https://helm.releases.hashicorp.com
+# helm repo update
+# kubectl apply -f clusters/noble/apps/vault/namespace.yaml
+# helm upgrade --install vault hashicorp/vault -n vault \
+#   --version 0.32.0 -f clusters/noble/apps/vault/values.yaml --wait --timeout 15m
+#
+# Post-install: initialize, store unseal key in Secret, apply optional unseal CronJob — see README.md
+#
+global:
+  tlsDisable: true
+
+injector:
+  enabled: true
+
+server:
+  enabled: true
+  dataStorage:
+    enabled: true
+    size: 10Gi
+    storageClass: longhorn
+    accessMode: ReadWriteOnce
+  ha:
+    enabled: false
+  standalone:
+    enabled: true
+    config: |
+      ui = true
+
+      listener "tcp" {
+        tls_disable = 1
+        address = "[::]:8200"
+        cluster_address = "[::]:8201"
+      }
+
+      storage "file" {
+        path = "/vault/data"
+      }
+
+  # Allow pod Ready before init/unseal so Helm --wait succeeds (see Vault /v1/sys/health docs).
+  readinessProbe:
+    enabled: true
+    path: "/v1/sys/health?uninitcode=204&sealedcode=204&standbyok=true"
+    port: 8200
+
+  # LAN: TLS terminates at Traefik + cert-manager; listener stays HTTP (global.tlsDisable).
+  ingress:
+    enabled: true
+    ingressClassName: traefik
+    annotations:
+      cert-manager.io/cluster-issuer: letsencrypt-prod
+    hosts:
+      - host: vault.apps.noble.lab.pcenicni.dev
+        paths: []
+    tls:
+      - secretName: vault-apps-noble-tls
+        hosts:
+          - vault.apps.noble.lab.pcenicni.dev
+
+ui:
+  enabled: true