home-server/clusters/noble/bootstrap/newt

Newt (Pangolin) — noble

This is the primary automation path for public hostnames to workloads in this cluster (it replaces in-cluster ExternalDNS). Newt is the on-prem agent that connects your cluster to a Pangolin site (WireGuard tunnel). The Fossorial Helm chart deploys one or more instances.

Secrets: Never commit endpoint, Newt ID, or Newt secret in plain YAML. If credentials were pasted into chat or CI logs, rotate them in Pangolin and recreate the Kubernetes Secret.

1. Create the Secret

Keys must match values.yaml (PANGOLIN_ENDPOINT, NEWT_ID, NEWT_SECRET).

Option A — SOPS (safe for GitOps)

Encrypt a normal Secret with Mozilla SOPS and age (see clusters/noble/secrets/README.md and .sops.yaml). The repo includes an encrypted example at clusters/noble/secrets/newt-pangolin-auth.secret.yaml — edit with sops after exporting SOPS_AGE_KEY_FILE to your age-key.txt, or create a new file and encrypt it.

export SOPS_AGE_KEY_FILE=/absolute/path/to/home-server/age-key.txt
sops clusters/noble/secrets/newt-pangolin-auth.secret.yaml
# then:
sops -d clusters/noble/secrets/newt-pangolin-auth.secret.yaml | kubectl apply -f -

Ansible (noble.yml) applies all clusters/noble/secrets/*.yaml automatically when age-key.txt exists at the repo root.

Option B — Imperative Secret (not in git)

kubectl apply -f clusters/noble/bootstrap/newt/namespace.yaml

kubectl -n newt create secret generic newt-pangolin-auth \
  --from-literal=PANGOLIN_ENDPOINT='https://pangolin.pcenicni.dev' \
  --from-literal=NEWT_ID='YOUR_NEWT_ID' \
  --from-literal=NEWT_SECRET='YOUR_NEWT_SECRET'

Use the Pangolin UI or Integration API (pick-site-defaults + create site) to obtain a Newt ID and secret for a new site if you are not reusing an existing pair.

2. Install the chart

helm repo add fossorial https://charts.fossorial.io
helm repo update
helm upgrade --install newt fossorial/newt \
  --namespace newt \
  --version 1.5.0 \
  -f clusters/noble/bootstrap/newt/values.yaml \
  --wait

3. DNS: CNAME at your DNS host + Pangolin API for routes

Pangolin does not replace your public DNS provider. Typical flow:

1. Link a domain in Pangolin (organization Domains). For CNAME-style domains, Pangolin shows the hostname you must CNAME to at Cloudflare / your registrar (see Domains).
2. Create public HTTP resources (and targets to your Newt site) via the Integration API — same flows as the UI. Swagger: https://<your-api-host>/v1/docs (self-hosted: enable enable_integration_api and route api.example.com → integration port per docs).

Minimal patterns (Bearer token = org or root API key):

export API_BASE='https://api.example.com/v1'   # your Pangolin Integration API base
export ORG_ID='your-org-id'
export TOKEN='your-integration-api-key'

# Domains already linked to the org (use domainId when creating a resource)
curl -sS -H "Authorization: Bearer ${TOKEN}" \
  "${API_BASE}/org/${ORG_ID}/domains"

# Create an HTTP resource on a domain (FQDN = subdomain + base domain for NS/wildcard domains)
curl -sS -X PUT -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' \
  "${API_BASE}/org/${ORG_ID}/resource" \
  -d '{
    "name": "Example app",
    "http": true,
    "domainId": "YOUR_DOMAIN_ID",
    "protocol": "tcp",
    "subdomain": "my-app"
  }'

# Point the resource at your Newt site backend (siteId from list sites / create site; ip:port inside the tunnel)
curl -sS -X PUT -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' \
  "${API_BASE}/resource/RESOURCE_ID/target" \
  -d '{
    "siteId": YOUR_SITE_ID,
    "ip": "10.x.x.x",
    "port": 443,
    "method": "http"
  }'

Exact JSON fields and IDs differ by domain type (ns vs cname vs wildcard); see Common API routes and Swagger.

4. Automate HTTP resources (Integration API + Ansible)

You still link domains in Pangolin and create CNAME records at your DNS host manually (Pangolin does not replace your registrar). After that, this repository can ensure public HTTP resources and Traefik targets exist for the same FQDNs you use in GitOps / Ansible:

• noble_authentik_ingress_extra_hosts (e.g. auth.example.com)
• noble_open_webui_public_host when set (e.g. webui.example.com)
• Optional extra list noble_pangolin_http_fqdns_extra in ansible/inventory/group_vars/all.yml

Steps:

0. Enable the Integration API in Pangolin’s config.yml on the Pangolin host — it is off by default. Add to config.yml:

   flags:
     enable_integration_api: true
   server:
     integration_port: 3003   # default; omit to keep 3003
   

   Then expose it with a Traefik route in config/traefik/dynamic_config.yml. The Integration API is a separate process from the main Pangolin server and needs its own hostname (e.g. api.pcenicni.dev):

   routers:
     int-api-router:
       rule: "Host(`api.pcenicni.dev`)"
       service: int-api-service
       entryPoints: [websecure]
       tls: { certResolver: letsencrypt }
     int-api-router-redirect:
       rule: "Host(`api.pcenicni.dev`)"
       service: int-api-service
       entryPoints: [web]
       middlewares: [redirect-to-https]
   services:
     int-api-service:
       loadBalancer:
         servers: [{ url: "http://pangolin:3003" }]
   

   After restarting Pangolin, verify: curl https://api.pcenicni.dev/v1/ should return {"message":"Pangolin Integration API"}. Also add a CNAME for api.pcenicni.dev pointing at the same upstream as pangolin.pcenicni.dev.

   Common mistake: https://pangolin.pcenicni.dev/api/v1 is the session-based external API — it will always return 401 to Bearer tokens. The Integration API must have its own Traefik-exposed hostname.

1. In Pangolin, create an organization API key (Integration API docs) with permission to manage domains, resources, and targets. The API expects Authorization: Bearer {apiKeyId}.{apiKeySecret} — paste id.secret as a single string into NOBLE_PANGOLIN_API_TOKEN, or set NOBLE_PANGOLIN_API_KEY_ID + only the secret in NOBLE_PANGOLIN_API_TOKEN. For NOBLE_PANGOLIN_SITE_ID as a niceId slug, enable Site → List Sites (the script falls back to listing sites if Get Site returns 404).

2. Add to repository .env (never commit secrets): NOBLE_PANGOLIN_API_BASE is the Integration API hostname with /v1 suffix — e.g. https://api.pcenicni.dev/v1 (not https://pangolin.pcenicni.dev/api/v1). Also set NOBLE_PANGOLIN_ORG_ID, NOBLE_PANGOLIN_API_TOKEN (optionally NOBLE_PANGOLIN_API_KEY_ID), NOBLE_PANGOLIN_SITE_ID (numeric siteId or niceId). Optionally NOBLE_PANGOLIN_TRAEFIK_IP / NOBLE_PANGOLIN_TRAEFIK_PORT — if unset, Ansible uses kubectl to read the Traefik Service LoadBalancer IP. TLS: NOBLE_PANGOLIN_CA_BUNDLE or NOBLE_PANGOLIN_INSECURE_SKIP_TLS_VERIFY=true for self-signed APIs

3. Set noble_pangolin_sync_http_resources: true in ansible/inventory/group_vars/all.yml (or pass -e noble_pangolin_sync_http_resources=true).

4. Run ansible-playbook playbooks/noble.yml --tags newt (or a full noble.yml) with KUBECONFIG pointed at the cluster.

Implementation: clusters/noble/bootstrap/newt/scripts/sync_pangolin_http_resources.py (stdlib Python 3). Dry run:
python3 clusters/noble/bootstrap/newt/scripts/sync_pangolin_http_resources.py --env-file .env --fqdns auth.example.com,webui.example.com --traefik-ip 192.168.50.211 --dry-run

The script matches each FQDN to the longest linked baseDomain in Pangolin, creates the HTTP resource if missing, then adds a target (siteId + Traefik ip:port, method: http) if none matches. Pangolin’s API is still evolving — if a call fails, compare with Swagger for your deployment version.

.env vs shell: If NOBLE_PANGOLIN_API_TOKEN (or other NOBLE_PANGOLIN_*) is set in your shell to an empty or old value, older script versions could ignore .env. Current script overwrites os.environ from .env when --env-file is passed — unset stray exports if you still see 401.

Authentik on a public name

Use noble_authentik_ingress_extra_hosts (see ansible/roles/noble_authentik/README.md) so the Authentik Ingress (and cert-manager SANs) include your public FQDN, then create the Pangolin HTTP resource + target to the same Traefik :443 endpoint as other apps. One Newt site can carry many hostnames.

What to put in Pangolin (resource + target)

1. Public hostname — the FQDN users type in the browser (must match noble_authentik_ingress_extra_hosts and your CNAME at the DNS host Pangolin documents for that domain).

2. Site — the Pangolin site that owns your Newt pair (same NEWT_ID / NEWT_SECRET as the cluster Secret). In the UI: Sites → pick the site connected to this cluster.

3. Target ip — an address reachable from inside the tunnel to Traefik HTTPS. On noble this is usually the Traefik LoadBalancer IP (repo pins 192.168.50.211 in clusters/noble/bootstrap/traefik/values.yaml). Confirm live:

   kubectl -n traefik get svc -l app.kubernetes.io/name=traefik -o wide

   Use EXTERNAL-IP (or LOAD_BALANCER_IP from the Service status) as ip. If Newt runs in the cluster, that MetalLB/LAN VIP is correct; if you run Newt elsewhere, use whatever L3 path reaches Traefik from that host.

4. Target port — 443 (TLS to Traefik; SNI carries the public hostname).

5. Target method — http in the Integration API examples above (TLS is still terminated at Traefik; Pangolin’s field names follow their docs).

Discovery in Pangolin’s UI: Domains (see required CNAME) → Resources → Add HTTP resource for the subdomain/FQDN → Targets / Backends → attach site + ip:port. Official flow: Domains, Integration API, and your deployment’s Swagger at https://<integration-api-host>/v1/docs when enabled.

LAN vs internet

• LAN / VPN: point *.apps.noble.lab.pcenicni.dev at the Traefik LoadBalancer (192.168.50.211) with local or split-horizon DNS if you want direct in-lab access.
• Internet-facing: use Pangolin resources + targets to the Newt site; public names rely on CNAME records at your DNS provider per Pangolin’s domain setup, not on ExternalDNS in the cluster.