home-server/clusters/noble/bootstrap/headlamp/README.md

Headlamp (noble)

Headlamp web UI for the cluster. Exposed on https://headlamp.apps.noble.lab.pcenicni.dev via Traefik + cert-manager (letsencrypt-prod), same pattern as Grafana.

• Chart: headlamp/headlamp 0.42.0 (config.sessionTTL: null still omits -session-ttl if needed — issue #4883)
• Namespace: headlamp
• OIDC TLS: cacert.pem (Mozilla bundle from curl CA extract) is baked into ConfigMap headlamp-oidc-ca-bundle via kustomization.yaml and mounted at /etc/ssl/headlamp/oidc-ca-bundle.pem for -oidc-ca-file (stops empty-PEM log noise; refresh the file occasionally). If Authentik used a private CA, append that PEM to cacert.pem (or replace) before sync.

Install

helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
helm repo update
kubectl apply -f clusters/noble/bootstrap/headlamp/namespace.yaml
helm upgrade --install headlamp headlamp/headlamp -n headlamp \
  --version 0.42.0 -f clusters/noble/bootstrap/headlamp/values.yaml --wait --timeout 10m

Sign-in uses a ServiceAccount token (Headlamp docs: create a limited SA for day-to-day use). This repo binds the Headlamp workload SA to the built-in edit ClusterRole (clusterRoleBinding.clusterRoleName: edit in values.yaml) — not cluster-admin. For cluster-scoped admin work, use kubectl with your admin kubeconfig. Optional OIDC in config.oidc replaces token login for SSO. In-cluster OIDC requires kube-apiserver OIDC (same Authentik app issuer + oidc-client-id: headlamp) or proxied K8s calls return 401 while /me still returns 200 — see talos/talconfig.yaml, oidc-noble-admins-clusterrolebinding.yaml, and ansible/roles/noble_authentik/README.md troubleshooting.

Sign-in token (ServiceAccount headlamp)

Use a short-lived token (Kubernetes 1.24+; requires permission to create TokenRequests):

export KUBECONFIG=/path/to/talos/kubeconfig   # or your admin kubeconfig
kubectl -n headlamp create token headlamp --duration=48h

Paste the printed JWT into Headlamp’s token field at https://headlamp.apps.noble.lab.pcenicni.dev.

OIDC: still “Unauthorized” while pod logs look fine

Headlamp logs like “Request completed successfully” for /plugins or static assets do not prove cluster API auth. After SSO, calls such as /clusters/main/version or …/selfsubjectrulesreviews use your OIDC id_token; kube-apiserver must validate it (Kubernetes OIDC).

1. Confirm API server flags match talos/talconfig.yaml (same oidc-issuer-url and oidc-client-id: headlamp as Secret headlamp-oidc / Authentik app headlamp). On Talos, apply regenerated control-plane machine configs and roll nodes so kube-apiserver actually picks up oidc-* extraArgs.
2. Inspect the id_token (browser devtools → Headlamp storage / network, or Authentik “Preview”): aud must include headlamp; for this repo’s oidc-noble-admins-clusterrolebinding.yaml, groups must list noble-admins exactly (if missing, see noble_authentik_headlamp_oidc_scopes and ansible/roles/noble_authentik/README.md).
3. API server logs often spell out the failure (invalid bearer token, wrong audience, unknown issuer). Check kube-apiserver logs on a control-plane node if steps 1–2 look correct.
4. oidc: email not verified: with oidc-username-claim: email, the API server rejects email_verified: false. Either set oidc-username-claim to a non-email claim (this repo uses preferred_username in talos/talconfig.yaml) or make Authentik issue email_verified: true for that user.

OIDC: no nodes, no CPU/memory, plugins misbehave

In-cluster Headlamp calls the API as your OIDC user, not as the headlamp ServiceAccount. The built-in edit role does not cover metrics.k8s.io or cluster nodes. Re-apply kubectl apply -k clusters/noble/bootstrap/headlamp so metrics-clusterrolebinding.yaml stays current: it binds noble-admins to headlamp-metrics-reader, which adds metrics, nodes, and read-only CustomResourceDefinitions (helps many plugins). Ensure metrics-server (or equivalent) is installed. If the plugin marketplace never loads, check the browser network tab for blocked HTTPS requests to external hosts.

To use another duration (cluster spec.serviceAccount / admission limits may cap it):

kubectl -n headlamp create token headlamp --duration=8760h