260 lines
15 KiB
Markdown
260 lines
15 KiB
Markdown
# Ansible getting started — Proxmox → Talos → cluster → Argo CD
|
||
|
||
This guide walks through the **intended order** for this repository: prepare **Proxmox VE** hosts and optionally form a **Proxmox cluster**, bring up **Talos** nodes and the **Kubernetes** control plane, install the **platform stack** with Ansible, then hand ongoing **bootstrap** configuration to **Argo CD** when you are ready.
|
||
|
||
Shorter reference tables and variable lists live in [`ansible/README.md`](../ansible/README.md). Deep operational detail for Talos and the noble lab checklist are in [`talos/README.md`](../talos/README.md) and [`talos/CLUSTER-BUILD.md`](../talos/CLUSTER-BUILD.md). Argo-specific sequencing is in [`clusters/noble/bootstrap/argocd/README.md`](../clusters/noble/bootstrap/argocd/README.md).
|
||
|
||
---
|
||
|
||
## What runs where
|
||
|
||
| Layer | What automates it | Where it runs |
|
||
|--------|-------------------|----------------|
|
||
| Proxmox hosts (repos, keys, upgrades, `pvecm`) | `proxmox_*.yml` playbooks | SSH to `proxmox_hosts` in `ansible/inventory/proxmox.yml` |
|
||
| Talos machine config, bootstrap, admin kubeconfig | `playbooks/talos_phase_a.yml` (or `deploy.yml` first half) | **Localhost** — needs LAN/VPN to node IPs (`:50000`, later `:6443`) |
|
||
| CNI, storage, ingress, cert-manager, Argo CD install, observability, policy, … | `playbooks/noble.yml` (or `deploy.yml` second half) | **Localhost** — uses `kubectl` / Helm against `KUBECONFIG` |
|
||
| Post-install reminders | `playbooks/post_deploy.yml` | Localhost |
|
||
|
||
Default Ansible inventory for Talos/noble playbooks is [`ansible/inventory/localhost.yml`](../ansible/inventory/localhost.yml) (`ansible.cfg` points there). **Proxmox** playbooks use **`-i inventory/proxmox.yml`** explicitly.
|
||
|
||
---
|
||
|
||
## Prerequisites (all phases)
|
||
|
||
On the machine that runs Ansible (your workstation or a bastion):
|
||
|
||
- **Ansible** (version compatible with the playbooks in this repo).
|
||
- **SSH** access to Proxmox hosts when running Proxmox playbooks.
|
||
- For Talos and Kubernetes phases: **same L2/L3 path** to lab node IPs (and eventually the API VIP) as documented in [`talos/README.md`](../talos/README.md) §3.
|
||
- **Talos tooling:** `talosctl` (version aligned with the node image), **`talhelper`**, **`kubectl`**, **`helm`**.
|
||
|
||
Optional but common for this repo:
|
||
|
||
- **SOPS** + **age** if you use encrypted manifests under `clusters/noble/secrets/` (see `clusters/noble/secrets/README.md`).
|
||
- Repository root **`.env`** copied from [`.env.sample`](../.env.sample) for cert-manager (Cloudflare DNS-01) and other optional components.
|
||
|
||
---
|
||
|
||
## 1. Proxmox — hosts and VE cluster
|
||
|
||
These steps are **independent** of Talos and Kubernetes. They configure community repositories, routine upgrades, SSH keys for `root`, and optionally create a **Proxmox VE cluster** (`pvecm`).
|
||
|
||
### 1.1 Inventory and variables
|
||
|
||
1. Copy the example inventory:
|
||
|
||
```bash
|
||
cp ansible/inventory/proxmox.example.yml ansible/inventory/proxmox.yml
|
||
```
|
||
|
||
2. Edit `ansible/inventory/proxmox.yml`: set `ansible_host`, `ansible_user` (typically `root`), and for the first login without key auth either **`--ask-pass`** or `ansible_password` (prefer **Ansible Vault** for passwords).
|
||
|
||
3. Edit [`ansible/inventory/group_vars/proxmox_hosts.yml`](../ansible/inventory/group_vars/proxmox_hosts.yml):
|
||
|
||
- **`proxmox_cluster_name`** — name for a **new** Proxmox cluster (changing it later does not rename an existing cluster).
|
||
- **`proxmox_cluster_master`** — inventory host name of the first node that runs `pvecm create`; leave empty only if the default ordering matches your intent (see role defaults).
|
||
- **`proxmox_root_authorized_key_files`** — public keys installed for `root` (after prepare, password login is usually unnecessary).
|
||
- **`proxmox_cluster_master_root_password`** — only if `pvecm add` still needs the master’s root password for joins; store with Vault in real environments.
|
||
|
||
Repo variables for Debian codename and subscription notices are already set in that file; adjust **`proxmox_repo_debian_codename`** if your PVE major tracks a different Debian base.
|
||
|
||
### 1.2 Run order
|
||
|
||
From the **`ansible/`** directory, targeting Proxmox:
|
||
|
||
```bash
|
||
cd ansible
|
||
|
||
# First contact: often need --ask-pass until SSH keys are installed
|
||
ansible-playbook -i inventory/proxmox.yml playbooks/proxmox_prepare.yml --ask-pass
|
||
|
||
ansible-playbook -i inventory/proxmox.yml playbooks/proxmox_upgrade.yml
|
||
ansible-playbook -i inventory/proxmox.yml playbooks/proxmox_cluster.yml
|
||
```
|
||
|
||
**`proxmox_prepare.yml`** runs the `proxmox_baseline` role (community repos, suppress no-subscription UI nag, install your **`authorized_keys`** for root). **`proxmox_upgrade.yml`** runs maintenance (dist-upgrade, cleanup, reboot when required) **serially** one host at a time. **`proxmox_cluster.yml`** bootstraps or joins the Proxmox cluster **serially**.
|
||
|
||
Convenience wrapper — same three steps in order:
|
||
|
||
```bash
|
||
ansible-playbook -i inventory/proxmox.yml playbooks/proxmox_ops.yml
|
||
```
|
||
|
||
After this phase you should have stable **Proxmox** hosts (and optionally a single **Proxmox cluster**) for creating the Talos VMs or bare-metal install targets. Creating those VMs or ISO boot entries is **outside** these playbooks; align disks and networks with [`talos/talconfig.yaml`](../talos/talconfig.yaml) and [`talos/CLUSTER-BUILD.md`](../talos/CLUSTER-BUILD.md) inventory.
|
||
|
||
---
|
||
|
||
## 2. Talos — secrets, generated configs, and Phase A
|
||
|
||
Talos automation assumes **`talos/talconfig.yaml`** (and secrets) describe your nodes. Ansible **does not** replace reading [`talos/README.md`](../talos/README.md): order matters (**genconfig → apply all nodes → bootstrap**), and **`--insecure`** is only for maintenance-mode APIs.
|
||
|
||
### 2.1 One-time (or when rotating machine config)
|
||
|
||
From the **`talos/`** directory:
|
||
|
||
```bash
|
||
cd talos
|
||
talhelper gensecret > talsecret.yaml
|
||
# talhelper validate talconfig talconfig.yaml # after edits
|
||
```
|
||
|
||
Do **not** commit `talsecret.yaml`, `talos/out/`, or `talos/kubeconfig`. If you use SOPS for secrets, follow talhelper and repo docs for encrypted variants.
|
||
|
||
### 2.2 Automated Phase A (recommended)
|
||
|
||
[`ansible/playbooks/talos_phase_a.yml`](../ansible/playbooks/talos_phase_a.yml) runs the **`talos_phase_a`** role on **localhost**:
|
||
|
||
1. **`talhelper genconfig -o out`** (when `noble_talos_genconfig` is true).
|
||
2. **`talosctl apply-config`** for each entry in **`noble_talos_nodes`** (maintenance vs secure mode is auto-probed unless you force `noble_talos_apply_mode`).
|
||
3. **`talosctl bootstrap`** on the bootstrap node (unless `noble_talos_skip_bootstrap` is true or etcd is already initialized).
|
||
4. **`talosctl kubeconfig`** writing **`talos/kubeconfig`** at the repo root (path set in the playbook).
|
||
|
||
Run from **`ansible/`**:
|
||
|
||
```bash
|
||
cd ansible
|
||
ansible-playbook playbooks/talos_phase_a.yml
|
||
```
|
||
|
||
Override IPs, machine filenames, or timing when your lab differs from [`ansible/roles/talos_phase_a/defaults/main.yml`](../ansible/roles/talos_phase_a/defaults/main.yml), for example:
|
||
|
||
```bash
|
||
ansible-playbook playbooks/talos_phase_a.yml \
|
||
-e 'noble_talos_bootstrap_node_ip=192.168.50.20' \
|
||
-e 'noble_talos_kubeconfig_endpoint=192.168.50.20'
|
||
```
|
||
|
||
If etcd is already bootstrapped and you only need apply/kubeconfig:
|
||
|
||
```bash
|
||
ansible-playbook playbooks/talos_phase_a.yml -e 'noble_talos_skip_bootstrap=true'
|
||
```
|
||
|
||
**Legacy:** `playbooks/talos_bootstrap.yml` only runs genconfig via `talos_bootstrap`; prefer **`talos_phase_a.yml`** for a full bring-up.
|
||
|
||
### 2.3 Sanity checks before the platform playbook
|
||
|
||
- **`talos/kubeconfig`** exists (or export **`KUBECONFIG`** to your own path).
|
||
- From the same network path you will use for Helm: **`kubectl get --raw /healthz`** returns **`ok`** (see [`talos/README.md`](../talos/README.md) §3 if the kubeconfig points at a VIP you cannot reach — use `noble_k8s_api_server_override` on **`noble.yml`** as in [`ansible/inventory/group_vars/all.yml`](../ansible/inventory/group_vars/all.yml)).
|
||
|
||
---
|
||
|
||
## 3. Kubernetes cluster creation — platform install (`noble.yml`)
|
||
|
||
Here “cluster creation” means: **empty Talos nodes are now members of a Kubernetes cluster**, and you are installing **CNI, storage, load balancing, ingress, cert-manager, GitOps, observability, policy**, and related components from this repo’s Helm/kubectl roles.
|
||
|
||
[`ansible/playbooks/noble.yml`](../ansible/playbooks/noble.yml) is the main playbook. It sets **`KUBECONFIG`** from the environment or defaults to **`$REPO_ROOT/talos/kubeconfig`**, runs an API **`/healthz`** preflight (with optional VIP fallback), then applies roles in dependency order (for example **Cilium** before **MetalLB** / **kube-vip**, **Kyverno** before **Longhorn** as documented in the playbook comments).
|
||
|
||
### 3.1 Secrets and feature flags
|
||
|
||
- **`.env`** at repo root: at minimum **`CLOUDFLARE_DNS_API_TOKEN`** when [`noble_cert_manager_require_cloudflare_secret`](../ansible/inventory/group_vars/all.yml) is true, so cert-manager can create DNS-01 issuers.
|
||
- **[`ansible/inventory/group_vars/all.yml`](../ansible/inventory/group_vars/all.yml)** toggles optional components (`noble_newt_install`, `noble_velero_install`, `noble_authentik_install`, Argo root Application flags, API server override, etc.).
|
||
|
||
### 3.2 Run
|
||
|
||
```bash
|
||
cd ansible
|
||
export KUBECONFIG=/absolute/path/to/home-server/talos/kubeconfig # optional if default path is correct
|
||
|
||
ansible-playbook playbooks/noble.yml
|
||
```
|
||
|
||
If the kubeconfig targets the API VIP but this host can only reach a control-plane IP:
|
||
|
||
```bash
|
||
ansible-playbook playbooks/noble.yml -e 'noble_k8s_api_server_override=https://192.168.50.20:6443'
|
||
```
|
||
|
||
Partial runs use **tags** (see [`ansible/README.md`](../ansible/README.md)).
|
||
|
||
### 3.3 One-shot pipeline from Talos through platform
|
||
|
||
[`ansible/playbooks/deploy.yml`](../ansible/playbooks/deploy.yml) imports **`talos_phase_a.yml`** then **`noble.yml`**. Use it when nodes are booted and reachable and you want a single command after updating `talconfig`:
|
||
|
||
```bash
|
||
cd ansible
|
||
ansible-playbook playbooks/deploy.yml
|
||
```
|
||
|
||
---
|
||
|
||
## 4. Cutover to Argo CD for deployment and config
|
||
|
||
Important mental model from [`clusters/noble/apps/README.md`](../clusters/noble/apps/README.md) and [`clusters/noble/bootstrap/argocd/README.md`](../clusters/noble/bootstrap/argocd/README.md):
|
||
|
||
- **Core platform** (CNI, storage, ingress, cert-manager, observability stack, Kyverno, etc.) is installed by **`noble.yml`** from **`clusters/noble/bootstrap/`** via Helm and kubectl — **Argo CD does not reconcile those core Helm charts by default** (those leaves live under **`argocd/app-of-apps/`** and are applied after Ansible Helm).
|
||
- **`noble-bootstrap-root`** tracks **`clusters/noble/bootstrap/`** (which **kustomize-includes** **`clusters/noble/apps/`**) for GitOps alignment with bootstrap kustomize and optional add-on **`Application`** manifests — enable **automated** sync only **after** Ansible has finished so Argo does not fight Helm mid-play.
|
||
|
||
### 4.1 What Ansible already does for Argo
|
||
|
||
At the **end** of **`noble.yml`**, after all Ansible Helm roles (**`noble_platform`**, **`noble_authentik`**, **`noble_velero`** when enabled), the play runs **`noble_argocd`** task file **`applications_post_platform.yml`**, which applies:
|
||
|
||
- **`bootstrap-root-application.yaml`** and **`kubectl apply -k clusters/noble/bootstrap/argocd/app-of-apps`** when **`noble_argocd_apply_bootstrap_root_application`** is true.
|
||
|
||
So the **bootstrap root Application CR** and **leaf Application** registrations typically already exist on the cluster after a successful **`noble.yml`**. They are created **last** on purpose so `argocd-application-controller` does not adopt resources before Helm installs them.
|
||
|
||
### 4.2 Before you enable GitOps automation
|
||
|
||
1. **Edit Git URLs** in **`bootstrap-root-application.yaml`**: set **`repoURL`** and **`targetRevision`** to your real remote and branch.
|
||
2. **Register the repository** in Argo CD (UI, `argocd repo add`, or a repository `Secret`) if it is private.
|
||
3. Leave **`noble-bootstrap-root`** on **manual** sync until Helm and the cluster match git (see **§5** in [`clusters/noble/bootstrap/argocd/README.md`](../clusters/noble/bootstrap/argocd/README.md)).
|
||
|
||
### 4.3 Enable automated sync for `noble-bootstrap-root`
|
||
|
||
After **`noble.yml`** completes successfully and you have refreshed the app in Argo, enable automated sync (prune + self-heal) using one of the methods documented in **§5** of [`clusters/noble/bootstrap/argocd/README.md`](../clusters/noble/bootstrap/argocd/README.md), for example:
|
||
|
||
```bash
|
||
kubectl patch application noble-bootstrap-root -n argocd --type merge \
|
||
-p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true},"syncOptions":["CreateNamespace=true"]}}}'
|
||
```
|
||
|
||
**Leaf** `Application` objects under **`clusters/noble/bootstrap/argocd/app-of-apps/`** remain **manual** until you intentionally turn on auto-sync **per chart** — when Argo should own a release, enable that leaf and **remove** the corresponding **`helm upgrade`** from Ansible so a single controller owns the release.
|
||
|
||
### 4.4 Optional apps repo path
|
||
|
||
Add only **additive** workloads under **`clusters/noble/apps/`** as `Application` manifests (see [`clusters/noble/apps/README.md`](../clusters/noble/apps/README.md)). **`kustomization.yaml`** may start empty; that is expected.
|
||
|
||
### 4.5 Post-deploy reminders
|
||
|
||
```bash
|
||
cd ansible
|
||
ansible-playbook playbooks/post_deploy.yml
|
||
```
|
||
|
||
This prints guidance about **SOPS** key handling and points back to the Argo README for sync policy.
|
||
|
||
### 4.6 Migrating from older Argo Application names
|
||
|
||
If you previously used different Application objects (for example a monolithic `noble-platform`), delete stale Applications as described in [`ansible/README.md`](../ansible/README.md) under **Migrating from Argo-managed `noble-platform`**, then re-apply the root manifests and reconcile with **`noble.yml`** if Helm drifted.
|
||
|
||
---
|
||
|
||
## Quick reference — minimal command sequence
|
||
|
||
```bash
|
||
# 1) Proxmox (from ansible/, with proxmox inventory)
|
||
ansible-playbook -i inventory/proxmox.yml playbooks/proxmox_ops.yml
|
||
|
||
# 2–3) Talos + Kubernetes platform (localhost inventory default)
|
||
cd ansible
|
||
ansible-playbook playbooks/deploy.yml
|
||
|
||
# 4) Reminders
|
||
ansible-playbook playbooks/post_deploy.yml
|
||
```
|
||
|
||
Then finish **Argo** cutover in the UI or CLI: register repo → refresh **`noble-bootstrap-root`** → enable **AUTO-SYNC** when ready → selectively enable leaf apps and retire overlapping Ansible Helm tasks.
|
||
|
||
---
|
||
|
||
## Related documentation
|
||
|
||
| Topic | Path |
|
||
|--------|------|
|
||
| Ansible overview | [`ansible/README.md`](../ansible/README.md) |
|
||
| Talos quick start | [`talos/README.md`](../talos/README.md) |
|
||
| Noble lab checklist | [`talos/CLUSTER-BUILD.md`](../talos/CLUSTER-BUILD.md) |
|
||
| Argo bootstrap and sync policy | [`clusters/noble/bootstrap/argocd/README.md`](../clusters/noble/bootstrap/argocd/README.md) |
|
||
| Optional Argo apps dir | [`clusters/noble/apps/README.md`](../clusters/noble/apps/README.md) |
|
||
| Deploy secrets | [`.env.sample`](../.env.sample) |
|