Update Ansible configuration to integrate SOPS for managing secrets. Enhance README.md with SOPS usage instructions and prerequisites. Remove External Secrets Operator references and related configurations from the bootstrap process, streamlining the deployment. Adjust playbooks and roles to apply SOPS-encrypted secrets automatically, improving security and clarity in secret management.
This commit is contained in:
Nikholas Pcenicni
@@ -1,6 +1,6 @@
|
||||
# Argo CD — optional applications (non-bootstrap)
|
||||
|
||||
**Base cluster configuration** (CNI, MetalLB, ingress, cert-manager, storage, observability stack, policy, Vault, etc.) is installed by **`ansible/playbooks/noble.yml`** from **`clusters/noble/bootstrap/`** — not from here.
|
||||
**Base cluster configuration** (CNI, MetalLB, ingress, cert-manager, storage, observability stack, policy, SOPS secrets path, etc.) is installed by **`ansible/playbooks/noble.yml`** from **`clusters/noble/bootstrap/`** — not from here.
|
||||
|
||||
**`noble-root`** (`clusters/noble/bootstrap/argocd/root-application.yaml`) points at **`clusters/noble/apps`**. Add **`Application`** manifests (and optional **`AppProject`** definitions) under this directory only for workloads that are additive and do not subsume the Ansible-managed platform.
|
||||
|
||||
|
||||
@@ -79,12 +79,6 @@ config:
|
||||
href: https://longhorn.apps.noble.lab.pcenicni.dev
|
||||
siteMonitor: http://longhorn-frontend.longhorn-system.svc.cluster.local:80
|
||||
description: Storage volumes, nodes, backups
|
||||
- Vault:
|
||||
icon: si-vault
|
||||
href: https://vault.apps.noble.lab.pcenicni.dev
|
||||
# Unauthenticated health (HEAD/GET) — not the redirecting UI root
|
||||
siteMonitor: http://vault.vault.svc.cluster.local:8200/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204
|
||||
description: Secrets engine UI (after init/unseal)
|
||||
- Velero:
|
||||
icon: mdi-backup-restore
|
||||
href: https://velero.io/docs/
|
||||
|
||||
@@ -52,7 +52,7 @@ Use **Settings → Repositories** in the UI, or `argocd repo add` / a `Secret` o
|
||||
|
||||
## 4. App-of-apps (optional GitOps only)
|
||||
|
||||
Bootstrap **platform** workloads (CNI, ingress, cert-manager, Kyverno, observability, Vault, etc.) are installed by
|
||||
Bootstrap **platform** workloads (CNI, ingress, cert-manager, Kyverno, observability, etc.) are installed by
|
||||
**`ansible/playbooks/noble.yml`** from **`clusters/noble/bootstrap/`** — not by Argo. **`clusters/noble/apps/kustomization.yaml`** is empty by default.
|
||||
|
||||
1. Edit **`root-application.yaml`**: set **`repoURL`** and **`targetRevision`** to this repository. The **`resources-finalizer.argocd.argoproj.io/background`** finalizer uses Argo’s path-qualified form so **`kubectl apply`** does not warn about finalizer names.
|
||||
|
||||
@@ -1,60 +0,0 @@
|
||||
# External Secrets Operator (noble)
|
||||
|
||||
Syncs secrets from external systems into Kubernetes **Secret** objects via **ExternalSecret** / **ClusterExternalSecret** CRDs.
|
||||
|
||||
- **Chart:** `external-secrets/external-secrets` **2.2.0** (app **v2.2.0**)
|
||||
- **Namespace:** `external-secrets`
|
||||
- **Helm release name:** `external-secrets` (matches the operator **ServiceAccount** name `external-secrets`)
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
helm repo add external-secrets https://charts.external-secrets.io
|
||||
helm repo update
|
||||
kubectl apply -f clusters/noble/bootstrap/external-secrets/namespace.yaml
|
||||
helm upgrade --install external-secrets external-secrets/external-secrets -n external-secrets \
|
||||
--version 2.2.0 -f clusters/noble/bootstrap/external-secrets/values.yaml --wait
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
kubectl -n external-secrets get deploy,pods
|
||||
kubectl get crd | grep external-secrets
|
||||
```
|
||||
|
||||
## Vault `ClusterSecretStore` (after Vault is deployed)
|
||||
|
||||
The checklist expects a **Vault**-backed store. Install Vault first (`talos/CLUSTER-BUILD.md` Phase E — Vault on Longhorn + auto-unseal), then:
|
||||
|
||||
1. Enable **KV v2** secrets engine and **Kubernetes** auth in Vault; create a **role** (e.g. `external-secrets`) that maps the cluster’s **`external-secrets` / `external-secrets`** service account to a policy that can read the paths you need.
|
||||
2. Copy **`examples/vault-cluster-secret-store.yaml`**, set **`spec.provider.vault.server`** to your Vault URL. This repo’s Vault Helm values use **HTTP** on port **8200** (`global.tlsDisable: true`): **`http://vault.vault.svc.cluster.local:8200`**. Use **`https://`** if you enable TLS on the Vault listener.
|
||||
3. If Vault uses a **private TLS CA**, configure **`caProvider`** or **`caBundle`** on the Vault provider — see [HashiCorp Vault provider](https://external-secrets.io/latest/provider/hashicorp-vault/). Do not commit private CA material to public git unless intended.
|
||||
4. Apply: **`kubectl apply -f …/vault-cluster-secret-store.yaml`**
|
||||
5. Confirm the store is ready: **`kubectl describe clustersecretstore vault`**
|
||||
|
||||
Example **ExternalSecret** (after the store is healthy):
|
||||
|
||||
```yaml
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: demo
|
||||
namespace: default
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: vault
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: demo-synced
|
||||
data:
|
||||
- secretKey: password
|
||||
remoteRef:
|
||||
key: secret/data/myapp
|
||||
property: password
|
||||
```
|
||||
|
||||
## Upgrades
|
||||
|
||||
Pin the chart version in `values.yaml` header comments; run the same **`helm upgrade --install`** with the new **`--version`** after reviewing [release notes](https://github.com/external-secrets/external-secrets/releases).
|
||||
@@ -1,31 +0,0 @@
|
||||
# ClusterSecretStore for HashiCorp Vault (KV v2) using Kubernetes auth.
|
||||
#
|
||||
# Do not apply until Vault is running, reachable from the cluster, and configured with:
|
||||
# - Kubernetes auth at mountPath (default: kubernetes)
|
||||
# - A role (below: external-secrets) bound to this service account:
|
||||
# name: external-secrets
|
||||
# namespace: external-secrets
|
||||
# - A policy allowing read on the KV path used below (e.g. secret/data/* for path "secret")
|
||||
#
|
||||
# Adjust server, mountPath, role, and path to match your Vault deployment. If Vault uses TLS
|
||||
# with a private CA, set provider.vault.caProvider or caBundle (see README).
|
||||
#
|
||||
# kubectl apply -f clusters/noble/bootstrap/external-secrets/examples/vault-cluster-secret-store.yaml
|
||||
---
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ClusterSecretStore
|
||||
metadata:
|
||||
name: vault
|
||||
spec:
|
||||
provider:
|
||||
vault:
|
||||
server: "http://vault.vault.svc.cluster.local:8200"
|
||||
path: secret
|
||||
version: v2
|
||||
auth:
|
||||
kubernetes:
|
||||
mountPath: kubernetes
|
||||
role: external-secrets
|
||||
serviceAccountRef:
|
||||
name: external-secrets
|
||||
namespace: external-secrets
|
||||
@@ -1,5 +0,0 @@
|
||||
# External Secrets Operator — apply before Helm.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: external-secrets
|
||||
@@ -1,10 +0,0 @@
|
||||
# External Secrets Operator — noble
|
||||
#
|
||||
# helm repo add external-secrets https://charts.external-secrets.io
|
||||
# helm repo update
|
||||
# kubectl apply -f clusters/noble/bootstrap/external-secrets/namespace.yaml
|
||||
# helm upgrade --install external-secrets external-secrets/external-secrets -n external-secrets \
|
||||
# --version 2.2.0 -f clusters/noble/bootstrap/external-secrets/values.yaml --wait
|
||||
#
|
||||
# CRDs are installed by the chart (installCRDs: true). Vault ClusterSecretStore: see README + examples/.
|
||||
commonLabels: {}
|
||||
@@ -8,13 +8,9 @@ resources:
|
||||
- kube-prometheus-stack/namespace.yaml
|
||||
- loki/namespace.yaml
|
||||
- fluent-bit/namespace.yaml
|
||||
- sealed-secrets/namespace.yaml
|
||||
- external-secrets/namespace.yaml
|
||||
- vault/namespace.yaml
|
||||
- newt/namespace.yaml
|
||||
- kyverno/namespace.yaml
|
||||
- velero/namespace.yaml
|
||||
- velero/longhorn-volumesnapshotclass.yaml
|
||||
- headlamp/namespace.yaml
|
||||
- grafana-loki-datasource/loki-datasource.yaml
|
||||
- vault/unseal-cronjob.yaml
|
||||
- vault/cilium-network-policy.yaml
|
||||
|
||||
@@ -35,7 +35,6 @@ x-kyverno-exclude-infra: &kyverno_exclude_infra
|
||||
- kube-node-lease
|
||||
- argocd
|
||||
- cert-manager
|
||||
- external-secrets
|
||||
- headlamp
|
||||
- kyverno
|
||||
- logging
|
||||
@@ -44,9 +43,7 @@ x-kyverno-exclude-infra: &kyverno_exclude_infra
|
||||
- metallb-system
|
||||
- monitoring
|
||||
- newt
|
||||
- sealed-secrets
|
||||
- traefik
|
||||
- vault
|
||||
|
||||
policyExclude:
|
||||
disallow-capabilities: *kyverno_exclude_infra
|
||||
|
||||
@@ -2,26 +2,24 @@
|
||||
|
||||
This is the **primary** automation path for **public** hostnames to workloads in this cluster (it **replaces** in-cluster ExternalDNS). [Newt](https://github.com/fosrl/newt) is the on-prem agent that connects your cluster to a **Pangolin** site (WireGuard tunnel). The [Fossorial Helm chart](https://github.com/fosrl/helm-charts) deploys one or more instances.
|
||||
|
||||
**Secrets:** Never commit endpoint, Newt ID, or Newt secret. If credentials were pasted into chat or CI logs, **rotate them** in Pangolin and recreate the Kubernetes Secret.
|
||||
**Secrets:** Never commit endpoint, Newt ID, or Newt secret in **plain** YAML. If credentials were pasted into chat or CI logs, **rotate them** in Pangolin and recreate the Kubernetes Secret.
|
||||
|
||||
## 1. Create the Secret
|
||||
|
||||
Keys must match `values.yaml` (`PANGOLIN_ENDPOINT`, `NEWT_ID`, `NEWT_SECRET`).
|
||||
|
||||
### Option A — Sealed Secret (safe for GitOps)
|
||||
### Option A — SOPS (safe for GitOps)
|
||||
|
||||
With the [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) controller installed (`clusters/noble/bootstrap/sealed-secrets/`), generate a `SealedSecret` from your workstation (rotate credentials in Pangolin first if they were exposed):
|
||||
Encrypt a normal **`Secret`** with [Mozilla SOPS](https://github.com/getsops/sops) and **age** (see **`clusters/noble/secrets/README.md`** and **`.sops.yaml`**). The repo includes an encrypted example at **`clusters/noble/secrets/newt-pangolin-auth.secret.yaml`** — edit with `sops` after exporting **`SOPS_AGE_KEY_FILE`** to your **`age-key.txt`**, or create a new file and encrypt it.
|
||||
|
||||
```bash
|
||||
chmod +x clusters/noble/bootstrap/sealed-secrets/examples/kubeseal-newt-pangolin-auth.sh
|
||||
export PANGOLIN_ENDPOINT='https://pangolin.pcenicni.dev'
|
||||
export NEWT_ID='YOUR_NEWT_ID'
|
||||
export NEWT_SECRET='YOUR_NEWT_SECRET'
|
||||
./clusters/noble/bootstrap/sealed-secrets/examples/kubeseal-newt-pangolin-auth.sh > newt-pangolin-auth.sealedsecret.yaml
|
||||
kubectl apply -f newt-pangolin-auth.sealedsecret.yaml
|
||||
export SOPS_AGE_KEY_FILE=/absolute/path/to/home-server/age-key.txt
|
||||
sops clusters/noble/secrets/newt-pangolin-auth.secret.yaml
|
||||
# then:
|
||||
sops -d clusters/noble/secrets/newt-pangolin-auth.secret.yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
Commit only the `.sealedsecret.yaml` file, not plain `Secret` YAML.
|
||||
**Ansible** (`noble.yml`) applies all **`clusters/noble/secrets/*.yaml`** automatically when **`age-key.txt`** exists at the repo root.
|
||||
|
||||
### Option B — Imperative Secret (not in git)
|
||||
|
||||
|
||||
@@ -1,50 +0,0 @@
|
||||
# Sealed Secrets (noble)
|
||||
|
||||
Encrypts `Secret` manifests so they can live in git; the controller decrypts **SealedSecret** resources into **Secret**s in-cluster.
|
||||
|
||||
- **Chart:** `sealed-secrets/sealed-secrets` **2.18.4** (app **0.36.1**)
|
||||
- **Namespace:** `sealed-secrets`
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
|
||||
helm repo update
|
||||
kubectl apply -f clusters/noble/bootstrap/sealed-secrets/namespace.yaml
|
||||
helm upgrade --install sealed-secrets sealed-secrets/sealed-secrets -n sealed-secrets \
|
||||
--version 2.18.4 -f clusters/noble/bootstrap/sealed-secrets/values.yaml --wait
|
||||
```
|
||||
|
||||
## Workstation: `kubeseal`
|
||||
|
||||
Install a **kubeseal** build compatible with the controller (match **app** minor, e.g. **0.36.x** for **0.36.1**). Examples:
|
||||
|
||||
- **Homebrew:** `brew install kubeseal` (check `kubeseal --version` against the chart’s `image.tag` in `helm show values`).
|
||||
- **GitHub releases:** [bitnami-labs/sealed-secrets](https://github.com/bitnami-labs/sealed-secrets/releases)
|
||||
|
||||
Fetch the cluster’s public seal cert (once per kube context):
|
||||
|
||||
```bash
|
||||
kubeseal --fetch-cert > /tmp/noble-sealed-secrets.pem
|
||||
```
|
||||
|
||||
Create a sealed secret from a normal secret manifest:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic example --from-literal=foo=bar --dry-run=client -o yaml \
|
||||
| kubeseal --cert /tmp/noble-sealed-secrets.pem -o yaml > example-sealedsecret.yaml
|
||||
```
|
||||
|
||||
Commit `example-sealedsecret.yaml`; apply it with `kubectl apply -f`. The controller creates the **Secret** in the same namespace as the **SealedSecret**.
|
||||
|
||||
**Noble example:** `examples/kubeseal-newt-pangolin-auth.sh` (Newt / Pangolin tunnel credentials).
|
||||
|
||||
## Backup the sealing key
|
||||
|
||||
If the controller’s private key is lost, existing sealed files cannot be decrypted on a new cluster. Back up the key secret after install:
|
||||
|
||||
```bash
|
||||
kubectl get secret -n sealed-secrets -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml > sealed-secrets-key-backup.yaml
|
||||
```
|
||||
|
||||
Store `sealed-secrets-key-backup.yaml` in a safe offline location (not in public git).
|
||||
@@ -1,19 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Emit a SealedSecret for newt-pangolin-auth (namespace newt).
|
||||
# Prerequisites: sealed-secrets controller running; kubeseal client (same minor as controller).
|
||||
# Rotate Pangolin/Newt credentials in the UI first if they were exposed, then set env vars and run:
|
||||
#
|
||||
# export PANGOLIN_ENDPOINT='https://pangolin.example.com'
|
||||
# export NEWT_ID='...'
|
||||
# export NEWT_SECRET='...'
|
||||
# ./kubeseal-newt-pangolin-auth.sh > newt-pangolin-auth.sealedsecret.yaml
|
||||
# kubectl apply -f newt-pangolin-auth.sealedsecret.yaml
|
||||
#
|
||||
set -euo pipefail
|
||||
kubectl apply -f "$(dirname "$0")/../../newt/namespace.yaml" >/dev/null 2>&1 || true
|
||||
kubectl -n newt create secret generic newt-pangolin-auth \
|
||||
--dry-run=client \
|
||||
--from-literal=PANGOLIN_ENDPOINT="${PANGOLIN_ENDPOINT:?}" \
|
||||
--from-literal=NEWT_ID="${NEWT_ID:?}" \
|
||||
--from-literal=NEWT_SECRET="${NEWT_SECRET:?}" \
|
||||
-o yaml | kubeseal -o yaml
|
||||
@@ -1,5 +0,0 @@
|
||||
# Sealed Secrets controller — apply before Helm.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: sealed-secrets
|
||||
@@ -1,18 +0,0 @@
|
||||
# Sealed Secrets — noble (Git-encrypted Secret workflow)
|
||||
#
|
||||
# helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
|
||||
# helm repo update
|
||||
# kubectl apply -f clusters/noble/bootstrap/sealed-secrets/namespace.yaml
|
||||
# helm upgrade --install sealed-secrets sealed-secrets/sealed-secrets -n sealed-secrets \
|
||||
# --version 2.18.4 -f clusters/noble/bootstrap/sealed-secrets/values.yaml --wait
|
||||
#
|
||||
# Client: install kubeseal (same minor as controller — see README).
|
||||
# Defaults are sufficient for the lab; override here if you need key renewal, resources, etc.
|
||||
#
|
||||
# GitOps pattern: create Secrets only via SealedSecret (or External Secrets + Vault).
|
||||
# Example (Newt): clusters/noble/bootstrap/sealed-secrets/examples/kubeseal-newt-pangolin-auth.sh
|
||||
# Backup the controller's sealing key: kubectl -n sealed-secrets get secret sealed-secrets-key -o yaml
|
||||
#
|
||||
# Talos cluster secrets (bootstrap token, cluster secret, certs) belong in talhelper talsecret /
|
||||
# SOPS — not Sealed Secrets. See talos/README.md.
|
||||
commonLabels: {}
|
||||
@@ -1,162 +0,0 @@
|
||||
# HashiCorp Vault (noble)
|
||||
|
||||
Standalone Vault with **file** storage on a **Longhorn** PVC (`server.dataStorage`). The listener uses **HTTP** (`global.tlsDisable: true`) for in-cluster use; add TLS at the listener when exposing outside the cluster.
|
||||
|
||||
- **Chart:** `hashicorp/vault` **0.32.0** (Vault **1.21.2**)
|
||||
- **Namespace:** `vault`
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
helm repo add hashicorp https://helm.releases.hashicorp.com
|
||||
helm repo update
|
||||
kubectl apply -f clusters/noble/bootstrap/vault/namespace.yaml
|
||||
helm upgrade --install vault hashicorp/vault -n vault \
|
||||
--version 0.32.0 -f clusters/noble/bootstrap/vault/values.yaml --wait --timeout 15m
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
kubectl -n vault get pods,pvc,svc
|
||||
kubectl -n vault exec -i sts/vault -- vault status
|
||||
```
|
||||
|
||||
## Cilium network policy (Phase G)
|
||||
|
||||
After **Cilium** is up, optionally restrict HTTP access to the Vault server pods (**TCP 8200**) to **`external-secrets`** and same-namespace clients:
|
||||
|
||||
```bash
|
||||
kubectl apply -f clusters/noble/bootstrap/vault/cilium-network-policy.yaml
|
||||
```
|
||||
|
||||
If you add workloads in other namespaces that call Vault, extend **`ingress`** in that manifest.
|
||||
|
||||
## Initialize and unseal (first time)
|
||||
|
||||
From a workstation with `kubectl` (or `kubectl exec` into any pod with `vault` CLI):
|
||||
|
||||
```bash
|
||||
kubectl -n vault exec -i sts/vault -- vault operator init -key-shares=1 -key-threshold=1
|
||||
```
|
||||
|
||||
**Lab-only:** `-key-shares=1 -key-threshold=1` keeps a single unseal key. For stronger Shamir splits, use more shares and store them safely.
|
||||
|
||||
Save the **Unseal Key** and **Root Token** offline. Then unseal once:
|
||||
|
||||
```bash
|
||||
kubectl -n vault exec -i sts/vault -- vault operator unseal
|
||||
# paste unseal key
|
||||
```
|
||||
|
||||
Or create the Secret used by the optional CronJob and apply it:
|
||||
|
||||
```bash
|
||||
kubectl -n vault create secret generic vault-unseal-key --from-literal=key='YOUR_UNSEAL_KEY'
|
||||
kubectl apply -f clusters/noble/bootstrap/vault/unseal-cronjob.yaml
|
||||
```
|
||||
|
||||
The CronJob runs every minute and unseals if Vault is sealed and the Secret is present.
|
||||
|
||||
## Auto-unseal note
|
||||
|
||||
Vault **OSS** auto-unseal uses cloud KMS (AWS, GCP, Azure, OCI), **Transit** (another Vault), etc. There is no first-class “Kubernetes Secret” seal. This repo uses an optional **CronJob** as a **lab** substitute. Production clusters should use a supported seal backend.
|
||||
|
||||
## Kubernetes auth (External Secrets / ClusterSecretStore)
|
||||
|
||||
**One-shot:** from the repo root, `export KUBECONFIG=talos/kubeconfig` and `export VAULT_TOKEN=…`, then run **`./clusters/noble/bootstrap/vault/configure-kubernetes-auth.sh`** (idempotent). Then **`kubectl apply -f clusters/noble/bootstrap/external-secrets/examples/vault-cluster-secret-store.yaml`** on its own line (shell comments **`# …`** on the same line are parsed as extra `kubectl` args and break `apply`). **`kubectl get clustersecretstore vault`** should show **READY=True** after a few seconds.
|
||||
|
||||
Run these **from your workstation** (needs `kubectl`; no local `vault` binary required). Use a **short-lived admin token** or the root token **only in your shell** — do not paste tokens into logs or chat.
|
||||
|
||||
**1. Enable the auth method** (skip if already done):
|
||||
|
||||
```bash
|
||||
kubectl -n vault exec -it sts/vault -- sh -c '
|
||||
export VAULT_ADDR=http://127.0.0.1:8200
|
||||
export VAULT_TOKEN="YOUR_ROOT_OR_ADMIN_TOKEN"
|
||||
vault auth enable kubernetes
|
||||
'
|
||||
```
|
||||
|
||||
**2. Configure `auth/kubernetes`** — the API **issuer** must match the `iss` claim on service account JWTs. With **kube-vip** / a custom API URL, discover it from the cluster (do not assume `kubernetes.default`):
|
||||
|
||||
```bash
|
||||
ISSUER=$(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer)
|
||||
REVIEWER=$(kubectl -n vault create token vault --duration=8760h)
|
||||
CA_B64=$(kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')
|
||||
```
|
||||
|
||||
Then apply config **inside** the Vault pod (environment variables are passed in with `env` so quoting stays correct):
|
||||
|
||||
```bash
|
||||
export VAULT_TOKEN="YOUR_ROOT_OR_ADMIN_TOKEN"
|
||||
export ISSUER REVIEWER CA_B64
|
||||
kubectl -n vault exec -i sts/vault -- env \
|
||||
VAULT_ADDR=http://127.0.0.1:8200 \
|
||||
VAULT_TOKEN="$VAULT_TOKEN" \
|
||||
CA_B64="$CA_B64" \
|
||||
REVIEWER="$REVIEWER" \
|
||||
ISSUER="$ISSUER" \
|
||||
sh -ec '
|
||||
echo "$CA_B64" | base64 -d > /tmp/k8s-ca.crt
|
||||
vault write auth/kubernetes/config \
|
||||
kubernetes_host="https://kubernetes.default.svc:443" \
|
||||
kubernetes_ca_cert=@/tmp/k8s-ca.crt \
|
||||
token_reviewer_jwt="$REVIEWER" \
|
||||
issuer="$ISSUER"
|
||||
'
|
||||
```
|
||||
|
||||
**3. KV v2** at path `secret` (skip if already enabled):
|
||||
|
||||
```bash
|
||||
kubectl -n vault exec -it sts/vault -- sh -c '
|
||||
export VAULT_ADDR=http://127.0.0.1:8200
|
||||
export VAULT_TOKEN="YOUR_ROOT_OR_ADMIN_TOKEN"
|
||||
vault secrets enable -path=secret kv-v2
|
||||
'
|
||||
```
|
||||
|
||||
**4. Policy + role** for the External Secrets operator SA (`external-secrets` / `external-secrets`):
|
||||
|
||||
```bash
|
||||
kubectl -n vault exec -it sts/vault -- sh -c '
|
||||
export VAULT_ADDR=http://127.0.0.1:8200
|
||||
export VAULT_TOKEN="YOUR_ROOT_OR_ADMIN_TOKEN"
|
||||
vault policy write external-secrets - <<EOF
|
||||
path "secret/data/*" {
|
||||
capabilities = ["read", "list"]
|
||||
}
|
||||
path "secret/metadata/*" {
|
||||
capabilities = ["read", "list"]
|
||||
}
|
||||
EOF
|
||||
vault write auth/kubernetes/role/external-secrets \
|
||||
bound_service_account_names=external-secrets \
|
||||
bound_service_account_namespaces=external-secrets \
|
||||
policies=external-secrets \
|
||||
ttl=24h
|
||||
'
|
||||
```
|
||||
|
||||
**5. Apply** **`clusters/noble/bootstrap/external-secrets/examples/vault-cluster-secret-store.yaml`** if you have not already, then verify:
|
||||
|
||||
```bash
|
||||
kubectl describe clustersecretstore vault
|
||||
```
|
||||
|
||||
See also [Kubernetes auth](https://developer.hashicorp.com/vault/docs/auth/kubernetes#configuration).
|
||||
|
||||
## TLS and External Secrets
|
||||
|
||||
`values.yaml` disables TLS on the Vault listener. The **`ClusterSecretStore`** example uses **`http://vault.vault.svc.cluster.local:8200`**. If you enable TLS on the listener, switch the URL to **`https://`** and configure **`caBundle`** or **`caProvider`** on the store.
|
||||
|
||||
## UI
|
||||
|
||||
Port-forward:
|
||||
|
||||
```bash
|
||||
kubectl -n vault port-forward svc/vault-ui 8200:8200
|
||||
```
|
||||
|
||||
Open `http://127.0.0.1:8200` and log in with the root token (rotate for production workflows).
|
||||
@@ -1,40 +0,0 @@
|
||||
# CiliumNetworkPolicy — restrict who may reach Vault HTTP listener (8200).
|
||||
# Apply after Cilium is healthy: kubectl apply -f clusters/noble/bootstrap/vault/cilium-network-policy.yaml
|
||||
#
|
||||
# Ingress-only policy: egress from Vault is unchanged (Kubernetes auth needs API + DNS).
|
||||
# Extend ingress rules if other namespaces must call Vault (e.g. app workloads).
|
||||
#
|
||||
# Ref: https://docs.cilium.io/en/stable/security/policy/language/
|
||||
---
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: vault-http-ingress
|
||||
namespace: vault
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: vault
|
||||
component: server
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
"k8s:io.kubernetes.pod.namespace": external-secrets
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: "8200"
|
||||
protocol: TCP
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
"k8s:io.kubernetes.pod.namespace": traefik
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: "8200"
|
||||
protocol: TCP
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
"k8s:io.kubernetes.pod.namespace": vault
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: "8200"
|
||||
protocol: TCP
|
||||
@@ -1,77 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Configure Vault Kubernetes auth + KV v2 + policy/role for External Secrets Operator.
|
||||
# Requires: kubectl (cluster access), jq optional (openid issuer); Vault reachable via sts/vault.
|
||||
#
|
||||
# Usage (from repo root):
|
||||
# export KUBECONFIG=talos/kubeconfig # or your path
|
||||
# export VAULT_TOKEN='…' # root or admin token — never commit
|
||||
# ./clusters/noble/bootstrap/vault/configure-kubernetes-auth.sh
|
||||
#
|
||||
# Then: kubectl apply -f clusters/noble/bootstrap/external-secrets/examples/vault-cluster-secret-store.yaml
|
||||
# Verify: kubectl describe clustersecretstore vault
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
: "${VAULT_TOKEN:?Set VAULT_TOKEN to your Vault root or admin token}"
|
||||
|
||||
ISSUER=$(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer)
|
||||
REVIEWER=$(kubectl -n vault create token vault --duration=8760h)
|
||||
CA_B64=$(kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')
|
||||
|
||||
kubectl -n vault exec -i sts/vault -- env \
|
||||
VAULT_ADDR=http://127.0.0.1:8200 \
|
||||
VAULT_TOKEN="$VAULT_TOKEN" \
|
||||
sh -ec '
|
||||
set -e
|
||||
vault auth list >/tmp/vauth.txt
|
||||
grep -q "^kubernetes/" /tmp/vauth.txt || vault auth enable kubernetes
|
||||
'
|
||||
|
||||
kubectl -n vault exec -i sts/vault -- env \
|
||||
VAULT_ADDR=http://127.0.0.1:8200 \
|
||||
VAULT_TOKEN="$VAULT_TOKEN" \
|
||||
CA_B64="$CA_B64" \
|
||||
REVIEWER="$REVIEWER" \
|
||||
ISSUER="$ISSUER" \
|
||||
sh -ec '
|
||||
echo "$CA_B64" | base64 -d > /tmp/k8s-ca.crt
|
||||
vault write auth/kubernetes/config \
|
||||
kubernetes_host="https://kubernetes.default.svc:443" \
|
||||
kubernetes_ca_cert=@/tmp/k8s-ca.crt \
|
||||
token_reviewer_jwt="$REVIEWER" \
|
||||
issuer="$ISSUER"
|
||||
'
|
||||
|
||||
kubectl -n vault exec -i sts/vault -- env \
|
||||
VAULT_ADDR=http://127.0.0.1:8200 \
|
||||
VAULT_TOKEN="$VAULT_TOKEN" \
|
||||
sh -ec '
|
||||
set -e
|
||||
vault secrets list >/tmp/vsec.txt
|
||||
grep -q "^secret/" /tmp/vsec.txt || vault secrets enable -path=secret kv-v2
|
||||
'
|
||||
|
||||
kubectl -n vault exec -i sts/vault -- env \
|
||||
VAULT_ADDR=http://127.0.0.1:8200 \
|
||||
VAULT_TOKEN="$VAULT_TOKEN" \
|
||||
sh -ec '
|
||||
vault policy write external-secrets - <<EOF
|
||||
path "secret/data/*" {
|
||||
capabilities = ["read", "list"]
|
||||
}
|
||||
path "secret/metadata/*" {
|
||||
capabilities = ["read", "list"]
|
||||
}
|
||||
EOF
|
||||
vault write auth/kubernetes/role/external-secrets \
|
||||
bound_service_account_names=external-secrets \
|
||||
bound_service_account_namespaces=external-secrets \
|
||||
policies=external-secrets \
|
||||
ttl=24h
|
||||
'
|
||||
|
||||
echo "Done. Issuer used: $ISSUER"
|
||||
echo ""
|
||||
echo "Next (each command on its own line — do not paste # comments after kubectl):"
|
||||
echo " kubectl apply -f clusters/noble/bootstrap/external-secrets/examples/vault-cluster-secret-store.yaml"
|
||||
echo " kubectl get clustersecretstore vault"
|
||||
@@ -1,5 +0,0 @@
|
||||
# HashiCorp Vault — apply before Helm.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: vault
|
||||
@@ -1,63 +0,0 @@
|
||||
# Optional lab auto-unseal: applies after Vault is initialized and Secret `vault-unseal-key` exists.
|
||||
#
|
||||
# 1) vault operator init -key-shares=1 -key-threshold=1 (lab only — single key)
|
||||
# 2) kubectl -n vault create secret generic vault-unseal-key --from-literal=key='YOUR_UNSEAL_KEY'
|
||||
# 3) kubectl apply -f clusters/noble/bootstrap/vault/unseal-cronjob.yaml
|
||||
#
|
||||
# OSS Vault has no Kubernetes/KMS seal; this CronJob runs vault operator unseal when the server is sealed.
|
||||
# Protect the Secret with RBAC; prefer cloud KMS auto-unseal for real environments.
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: vault-auto-unseal
|
||||
namespace: vault
|
||||
spec:
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 1
|
||||
failedJobsHistoryLimit: 3
|
||||
schedule: "*/1 * * * *"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 100
|
||||
runAsGroup: 1000
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: unseal
|
||||
image: hashicorp/vault:1.21.2
|
||||
imagePullPolicy: IfNotPresent
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
env:
|
||||
- name: VAULT_ADDR
|
||||
value: http://vault.vault.svc:8200
|
||||
command:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- |
|
||||
test -f /secrets/key || exit 0
|
||||
status="$(vault status -format=json 2>/dev/null || true)"
|
||||
echo "$status" | grep -q '"initialized":true' || exit 0
|
||||
echo "$status" | grep -q '"sealed":false' && exit 0
|
||||
vault operator unseal "$(cat /secrets/key)"
|
||||
volumeMounts:
|
||||
- name: unseal
|
||||
mountPath: /secrets
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: unseal
|
||||
secret:
|
||||
secretName: vault-unseal-key
|
||||
optional: true
|
||||
items:
|
||||
- key: key
|
||||
path: key
|
||||
@@ -1,62 +0,0 @@
|
||||
# HashiCorp Vault — noble (standalone, file storage on Longhorn; TLS disabled on listener for in-cluster HTTP).
|
||||
#
|
||||
# helm repo add hashicorp https://helm.releases.hashicorp.com
|
||||
# helm repo update
|
||||
# kubectl apply -f clusters/noble/bootstrap/vault/namespace.yaml
|
||||
# helm upgrade --install vault hashicorp/vault -n vault \
|
||||
# --version 0.32.0 -f clusters/noble/bootstrap/vault/values.yaml --wait --timeout 15m
|
||||
#
|
||||
# Post-install: initialize, store unseal key in Secret, apply optional unseal CronJob — see README.md
|
||||
#
|
||||
global:
|
||||
tlsDisable: true
|
||||
|
||||
injector:
|
||||
enabled: true
|
||||
|
||||
server:
|
||||
enabled: true
|
||||
dataStorage:
|
||||
enabled: true
|
||||
size: 10Gi
|
||||
storageClass: longhorn
|
||||
accessMode: ReadWriteOnce
|
||||
ha:
|
||||
enabled: false
|
||||
standalone:
|
||||
enabled: true
|
||||
config: |
|
||||
ui = true
|
||||
|
||||
listener "tcp" {
|
||||
tls_disable = 1
|
||||
address = "[::]:8200"
|
||||
cluster_address = "[::]:8201"
|
||||
}
|
||||
|
||||
storage "file" {
|
||||
path = "/vault/data"
|
||||
}
|
||||
|
||||
# Allow pod Ready before init/unseal so Helm --wait succeeds (see Vault /v1/sys/health docs).
|
||||
readinessProbe:
|
||||
enabled: true
|
||||
path: "/v1/sys/health?uninitcode=204&sealedcode=204&standbyok=true"
|
||||
port: 8200
|
||||
|
||||
# LAN: TLS terminates at Traefik + cert-manager; listener stays HTTP (global.tlsDisable).
|
||||
ingress:
|
||||
enabled: true
|
||||
ingressClassName: traefik
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
hosts:
|
||||
- host: vault.apps.noble.lab.pcenicni.dev
|
||||
paths: []
|
||||
tls:
|
||||
- secretName: vault-apps-noble-tls
|
||||
hosts:
|
||||
- vault.apps.noble.lab.pcenicni.dev
|
||||
|
||||
ui:
|
||||
enabled: true
|
||||
38
clusters/noble/secrets/README.md
Normal file
38
clusters/noble/secrets/README.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# SOPS-encrypted cluster secrets (noble)
|
||||
|
||||
Secrets that belong in git are stored here as **Mozilla SOPS** files encrypted with [age](https://github.com/FiloSottile/age). The matching **private** key lives in **`age-key.txt`** at the repository root (gitignored — create with `age-keygen -o age-key.txt` and add the public key to **`.sops.yaml`** if you rotate keys).
|
||||
|
||||
**Migrating from an older cluster** that ran **Vault**, **Sealed Secrets**, or **External Secrets Operator:** uninstall those Helm releases (`helm uninstall vault -n vault`, etc.), delete their namespaces if empty, and export any secrets you still need into plain **`Secret`** YAML here, then encrypt with **`sops`** before committing.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [sops](https://github.com/getsops/sops) and **age** on the machine that encrypts or applies secrets.
|
||||
|
||||
## Edit or create a Secret
|
||||
|
||||
```bash
|
||||
export SOPS_AGE_KEY_FILE=/absolute/path/to/home-server/age-key.txt
|
||||
|
||||
# Create a new file from a template, then encrypt:
|
||||
sops clusters/noble/secrets/example.secret.yaml
|
||||
|
||||
# Or edit an existing encrypted file (opens decrypted in $EDITOR):
|
||||
sops clusters/noble/secrets/newt-pangolin-auth.secret.yaml
|
||||
```
|
||||
|
||||
## Apply to the cluster
|
||||
|
||||
```bash
|
||||
export KUBECONFIG=/absolute/path/to/home-server/talos/kubeconfig
|
||||
export SOPS_AGE_KEY_FILE=/absolute/path/to/home-server/age-key.txt
|
||||
|
||||
sops -d clusters/noble/secrets/newt-pangolin-auth.secret.yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
**Ansible** (`noble.yml`) runs the same decrypt-and-apply step for every `*.yaml` in this directory when **`age-key.txt`** exists and **`noble_apply_sops_secrets`** is true (see `ansible/group_vars/all.yml`).
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `newt-pangolin-auth.secret.yaml` | Pangolin tunnel credentials for [Newt](../../bootstrap/newt/README.md) (`PANGOLIN_ENDPOINT`, `NEWT_ID`, `NEWT_SECRET`). Replace placeholders and re-encrypt before relying on them. |
|
||||
30
clusters/noble/secrets/newt-pangolin-auth.secret.yaml
Normal file
30
clusters/noble/secrets/newt-pangolin-auth.secret.yaml
Normal file
@@ -0,0 +1,30 @@
|
||||
apiVersion: ENC[AES256_GCM,data:FaA=,iv:EsqIdZmNS4hfzwCZ0gL7Q5Czaz8Bii3jWFu60lKmgVo=,tag:tfr4yUuTiH4s+ufYW/dpCA==,type:str]
|
||||
kind: ENC[AES256_GCM,data:ozpTcG9F,iv:Q1EZ896Plhyz2qM4JJRnBf940kbVLSwyIIPUcDGBZFA=,tag:1bWEgI+I4Ni5J70MlohYdA==,type:str]
|
||||
metadata:
|
||||
name: ENC[AES256_GCM,data:moXbGuT6ZOGhgVUBNcpHeLZQ,iv:1WDtxT/Et/6lxx1Mj93CQME8o0lhzxnBMkdSqP/n3R0=,tag:v+iqfE8tzCx8ZOMUW7OyEA==,type:str]
|
||||
namespace: ENC[AES256_GCM,data:33/AMg==,iv:M0GvB/70nHh4MVR1saZy1pGY8IFFzkzGdJl4szHJbCI=,tag:0+1LX/EnkAP0FZ6ARKZNAA==,type:str]
|
||||
type: ENC[AES256_GCM,data:3io5utU1,iv:QqMDNL/R8SR7TC9mwDdDd3V6VOo+csgeiZCr2AdOZjw=,tag:/KSMy+vNz7Qj/I463eG0LQ==,type:str]
|
||||
stringData:
|
||||
PANGOLIN_ENDPOINT: ENC[AES256_GCM,data:a/2QTnGYnNXGxNm8QSVTKC6I+r88J1m1CdMmTA==,iv:L2LvLD7IRX8wdAzALAWQ2ojB2OjWDIcVKrdi/lSvZFY=,tag:ALffRF9bncxA8CExSaRmHA==,type:str]
|
||||
NEWT_ID: ENC[AES256_GCM,data:Xfe8QvBdX62CciYXYwMfJAzIE/0=,iv:tA+FJ93tsjJ29L3bSxNAEooiKPMc+5pa00EpQ2cJkho=,tag:auiR/zQjnsmyllXbSJf3KA==,type:str]
|
||||
NEWT_SECRET: ENC[AES256_GCM,data:XY8XZOkZ+GpnjljbvtaH2oGJpDoZ47fN,iv:+J5sb7saqbVwHEyemx3CUSsdKArubRdPCLGbT09sFLM=,tag:zUowv8I1CaWZH+KLYOwKYw==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age1juym5p3ez3dkt0dxlznydgfgqvaujfnyk9hpdsssf50hsxeh3p4sjpf3gn
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0RWppdWxZUEYzc2I2TURi
|
||||
dm1pUzVaNDA4YldsWkFJODl1MWZ6MXFxWnhjCnVtU1VEQnJqbTI5M0hWM2FCaVlS
|
||||
aXprTm42bTlldUVHMmxpUUJiWEVhcXcKLS0tIGNLVnNtNDdMQ0VVeDV1N29nOW9F
|
||||
clhLa2tPdWtRMWYzc2YrR0hSQXczTlUK6hYj4HxQvu6Kqn/Ki+cYv9x5nvolyGqQ
|
||||
N4g9z+t6orT6MYseWPf0uyovC/5iOOC6z/2exVe7/0rYo7ZOFm6dYQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-03-29T23:37:33Z"
|
||||
mac: ENC[AES256_GCM,data:uKtdqJhwE4HLCenHH+RG8O2yfVIcGbiXznL9ouAXhDLnQh/ksgeczr2fyyn9hs/JhCozAqRrF8vnYZsIdfG1DQfHjXn6Ro6gzYC0YR+gvFU8Mz9uPdVX3HYjUrzKJ5GhhBami0USZtLdGKOGgFDYmFoDsD/PmMXLUol8qJdW8Uk=,iv:rIfQI17+3vNBB1n//D7Wnl/SLWFjV0pgZDteumlS2f8=,tag:xibCfJdZQS+aB75drmY1VA==,type:str]
|
||||
pgp: []
|
||||
unencrypted_suffix: _unencrypted
|
||||
version: 3.9.3
|
||||
Reference in New Issue
Block a user