guildhall/DEPLOY-EXPLORATORY-2026-04-21.md

Guildhall deploy exploratory — Talos/Hetzner cluster state

Date: 2026-04-21 Scope: Read-only audit of the Talos/Hetzner Kubernetes cluster to inform Guildhall's initial deployment. Method: kubectl against the cluster via ~/projects/substrate-project/guildhouse-talos-bootstrap/kubeconfig. No mutations. Takeaway (synthesis at end): Guildhall fits cleanly into the existing Keycloak/Forgejo deployment pattern: plain Deployment + Deployment-backed Postgres + Longhorn PVC + Hetzner LoadBalancer + Cloudflare-terminated TLS. No new infrastructure components required. The v1 substrate foundation (bascule / quartermaster / spire / chronicle / substrate-operator) is Flux-manifested but broken and not running; governance integration is explicitly follow-up work, not blocking.

---

1. Cluster basics

Control plane endpoint	https://178.104.100.159:6443
kubectl client	v1.32.2
kubectl server	v1.32.3
Nodes	5 (3 control-plane + 2 workers), all Ready
OS	Talos v1.9.5
Kernel	6.12.18-talos
Container runtime	containerd 2.0.3
Cluster age	10 days

gsh-cp-01        control-plane   10.0.1.10
gsh-cp-02        control-plane   10.0.1.20
gsh-cp-03        control-plane   10.0.1.21
gsh-worker-01    worker          10.0.1.22
gsh-worker-02    worker          10.0.1.30

Matches the memory-carried description (Hetzner Talos cluster 2026-04-11: Talos 1.9.5, 5 nodes, 3 CP + 2 worker). No drift.

2. Namespace inventory

Namespace	Purpose	Workloads
cert-manager	cert-manager 3 controllers (cert-manager, cainjector, webhook)	3 Deployments
flux-system	Flux GitOps	4 Deployments (source / kustomize / helm / notification controllers)
forgejo	Forgejo git (self-hosted)	Deployment + Postgres Deployment + Runner Deployment (0/1, stuck)
keycloak	Keycloak OIDC IdP	Deployment + Postgres Deployment
longhorn-system	Longhorn CSI storage	5 DaemonSets + 6 Deployments + UI
kube-system, kube-public, kube-node-lease	K8s system	—
default	Empty	—

Application workloads: forgejo, keycloak. These are the reference patterns for Guildhall.

3. Ingress / gateway state

No traditional ingress controller, no Gateway API:

• kubectl get ingressclasses → No resources found
• kubectl get gatewayclasses → server doesn't have the resource type (Gateway API CRDs not installed)
• No HAProxy / nginx / traefik / istio / kong pods anywhere

Traffic reaches services via type: LoadBalancer directly, backed by the Hetzner Cloud Controller Manager (hcloud-cloud-controller-manager running in kube-system). Each LoadBalancer Service provisions a real Hetzner Cloud Load Balancer via annotations:

annotations:
  load-balancer.hetzner.cloud/location: nbg1
  load-balancer.hetzner.cloud/name: keycloak-lb-v2
  load-balancer.hetzner.cloud/type: lb11
  load-balancer.hetzner.cloud/use-private-ip: "false"

Cilium Envoy DaemonSet pods exist on every node (cilium-envoy for L7 proxy features — CiliumNetworkPolicy L7 filtering, not Gateway API). enable-l7-proxy: true is set in the Cilium config.

Existing LoadBalancer services and their public addresses:

Service	Namespace	IPv4	IPv6	Ports
forgejo-http	forgejo	46.225.47.75	2a01:4f8:1c1f:65bb::1	80 → 30000, 22 → 30022
keycloak	keycloak	162.55.157.168	2a01:4f8:1c1d:1109::1	80 → 30080

Both expose port 80 only. No in-cluster TLS termination. TLS is terminated upstream at Cloudflare (git.guildhouse.dev and auth.guildhouse.dev resolve to Cloudflare IPs; Cloudflare proxies to the Hetzner LB IPs).

4. cert-manager / TLS

cert-manager is installed and healthy but no Certificate resources exist anywhere on the cluster.

ClusterIssuer	Status
letsencrypt-prod	Ready
letsencrypt-staging	Ready

Both ClusterIssuers are provisioned and ready to issue. They just aren't being used yet — TLS is currently handled via Cloudflare's Universal SSL / Full mode at the edge, with HTTP between Cloudflare and the Hetzner LBs.

Implication for Guildhall: can choose between the existing Cloudflare-termination pattern (simplest, matches forgejo/keycloak) or start using cert-manager (more work, cluster-integrated certs). The former is tonight's path; the latter is a hygiene follow-up.

5. Database patterns

No Postgres operator installed. CRDs checked (all absent):

• CloudNativePG (clusters.postgresql.cnpg.io)
• Zalando postgres-operator (postgresqls.acid.zalan.do)
• Crunchy PGO (postgresclusters.postgres-operator.crunchydata.com)

Existing pattern is plain Deployment + PVC:

forgejo-postgres    Deployment    postgres:16    PVC: forgejo-db  10Gi   longhorn
keycloak-postgres   Deployment    postgres:16    PVC: keycloak-db  5Gi   longhorn

Storage: Longhorn 1.x, single StorageClass longhorn (default). All PVCs use it. 5 DaemonSet replicas of longhorn-manager confirm storage is healthy across all nodes.

Current PVCs:

PVC	Namespace	Size	StorageClass
forgejo-data	forgejo	20Gi	longhorn
forgejo-db	forgejo	10Gi	longhorn
runner-cache	forgejo	5Gi	longhorn
keycloak-db	keycloak	5Gi	longhorn

6. Secrets management

None of the common secret managers are installed:

• External Secrets Operator: absent
• Sealed Secrets: absent
• SPIRE/SPIFFE: absent (Flux has a Kustomization for it but the spire-system namespace doesn't exist — see §10 Flux state)
• Vault: absent

Secrets are plain Opaque Secret resources. Examples:

• forgejo/forgejo-secrets (3 keys)
• keycloak/keycloak-secrets (2 keys)

Managed out-of-band (likely committed to the private Flux source repo or applied via kubectl during bootstrap). No rotation mechanism visible.

7. Existing workload patterns

Reference: keycloak Deployment (cleanest example — the only Flux Kustomization that's Ready):

• Image: quay.io/keycloak/keycloak:26.0 (public registry)
• Env composition: mix of literal value: (DB host, DB port, realm name) and valueFrom.secretKeyRef (admin password, DB password)
• Labels: app.kubernetes.io/name=keycloak, app.kubernetes.io/part-of=guildhouse
• Config files: ConfigMap-mounted realm import (keycloak-realm-import)
• Resources: resource requests/limits not aggressively set (defaults mostly)
• Service: type: LoadBalancer with Hetzner annotations, exposes port 80 only
• TLS: none in-cluster; Cloudflare upstream

Reference: forgejo-postgres Deployment:

• Image: postgres:16 (public Docker Hub)
• Env: POSTGRES_USER, POSTGRES_DB literal; POSTGRES_PASSWORD from Secret
• PGDATA: /var/lib/postgresql/data/pgdata (standard subdirectory to avoid lost+found issues)
• Volume: PVC mounted at /var/lib/postgresql/data

No existing Elixir/Phoenix deployment to reference. Guildhall will be the first. The pattern will follow the Keycloak/Forgejo shape applied to Phoenix's runtime requirements.

8. Guildhouse-specific components (v1 foundation)

Currently running: none.

• No pod matching substrate, bascule, chronicle, quartermaster, or spire across all namespaces. The v1 substrate foundation is absent from the cluster's running state.
• Flux has Kustomizations for bascule, quartermaster, spire, automation, governance-talos, gitops-controller — all failing on a dependency chain:

spire           → fails: namespace "spire-system" does not exist
quartermaster   → fails: dependency flux-system/spire is not ready
bascule         → fails: dependency flux-system/quartermaster is not ready
automation      → fails: dependency flux-system/quartermaster is not ready
gitops-controller → fails: dependency flux-system/quartermaster is not ready
governance-talos  → fails: dependency flux-system/cluster-infra is not ready
cluster-infra   → SUSPENDED + YAML decode error on 10-cilium-values.yaml

This chain needs to be unblocked for the v1 substrate foundation to reach the cluster, but this is explicitly NOT Guildhall's blocker. Guildhall is the standalone orchestration/presentation layer; it composes with substrate via CRD watches once substrate is running, but doesn't require substrate present to stand up and serve its web UI.

9. Networking specifics

Cilium version: v1.16.5 (Cilium 1.16 series, recent but not 1.17-cutting-edge)

Key Cilium config (from kube-system/cilium-config):

Flag	Value	Notes
kube-proxy-replacement	true	Cilium replaces kube-proxy (full eBPF mode)
enable-ipv4	true	IPv4 on pod network
enable-ipv6	false	IPv6 NOT enabled at pod network (LBs get Hetzner-assigned v6 externally)
enable-l7-proxy	true	Envoy DaemonSet for L7 filtering
enable-hubble	true	Hubble observability
ipam	kubernetes	Host-IPAM, not cluster-pool

Not enabled / not present:

• BGP control plane (ciliumbgppeeringpolicies CRD absent)
• L2 announcements (ciliuml2announcementpolicies CRD present but zero resources)
• LoadBalancerIPPool (CRD present but zero resources — Hetzner CCM handles LB IPs instead)
• Gateway API (gatewayclasses CRD absent)
• ClusterMesh (single-cluster)

NetworkPolicies in place (only 3, all in flux-system):

• allow-egress
• allow-scraping
• allow-webhooks (scoped to app=notification-controller)

CiliumNetworkPolicies: none. Workloads rely on default-allow between pods. Guildhall deployment can proceed without adding policies; adding them is hardening follow-up.

10. Deployment automation

GitOps: Flux is the sole mechanism. Running components:

• source-controller, kustomize-controller, helm-controller, notification-controller — all 1/1 Ready

Sources: one GitRepository registered:

flux-system / guildhouse-deploy
  URL:    https://github.com/gh-tking/guildhouse-deploy-talos-mirror
  STATUS: Ready (artifact stored for main@169e077f)

Kustomizations: 9 total, summary:

Name	Status
keycloak	✅ Ready (applied revision 169e077f)
forgejo	❌ health check failed (forgejo-runner Deployment stuck InProgress)
cluster-infra	❌ SUSPENDED + YAML decode error
spire	❌ spire-system namespace not found
quartermaster	❌ depends on spire (not ready)
bascule	❌ depends on quartermaster (not ready)
automation	❌ depends on quartermaster (not ready)
gitops-controller	❌ depends on quartermaster (not ready)
governance-talos	❌ depends on cluster-infra (not ready)

Key observation: only keycloak flows through Flux successfully. Everything else is either suspended, blocked on missing upstream dependencies, or has a YAML error in the source repo.

HelmRepositories and HelmReleases: none.

Changes land on the cluster: currently via Flux against the GitHub-hosted source repo for the one working Kustomization (keycloak), otherwise via direct kubectl apply (given the broken Flux chain).

---

Synthesis

What Guildhall can leverage

• Longhorn StorageClass — works out of the box for Postgres PVC. 5Gi is ample for initial Guildhall DB (matches keycloak-db sizing).
• Hetzner CCM LoadBalancer — a LoadBalancer Service with load-balancer.hetzner.cloud/* annotations provisions a new Hetzner LB automatically. Cost is ~€5/mo for an lb11 tier. Matches forgejo / keycloak exactly.
• Cloudflare-at-the-edge TLS — DNS at guildhall.guildhouse.dev points at the Hetzner LB IPv4, Cloudflare terminates TLS, origin is plain HTTP on port 80. Zero cert-manager work required for v1.
• Keycloak as OIDC IdP — already running at auth.guildhouse.dev. When Guildhall wires its OIDC config (currently commented out in config/runtime.exs), the endpoint is ready. Not blocking tonight.
• cert-manager ClusterIssuers — letsencrypt-prod and letsencrypt-staging are ready, available as upgrade-path from Cloudflare-edge TLS to cluster-terminated TLS if/when that hygiene pass happens.
• Reference deployment pattern — keycloak's Deployment shape (public image, env-from-secret, ConfigMap for data, Service type=LoadBalancer, Postgres sibling Deployment + PVC) maps directly to Guildhall. Apply the same template.
• Flux GitOps pipeline exists (if desired) — a new Kustomization in guildhouse-deploy-talos-mirror for Guildhall would auto-deploy. BUT the Flux state is currently messy — most Kustomizations are broken — so a direct kubectl apply path is cleaner for the v1 Guildhall deploy, with a follow-up Flux migration once the broader chain is healed.

What Guildhall needs that the cluster doesn't have yet

• Guildhall container image. Must be built locally via mix release + Dockerfile and pushed to a registry the cluster can pull from. Registry options:
  • ghcr.io/gh-tking/guildhall:<tag> — public GitHub Container Registry (requires packaging via the GitHub Actions or manual docker push)
  • Docker Hub under a personal account
  • Forgejo container registry at git.guildhouse.dev/tking/guildhall:<tag> — Forgejo 1.19+ supports OCI registry; this is the most consistent choice with the rest of the Guildhouse tooling
  • A private Hetzner-region ghcr mirror
• Secrets: guildhall-secrets Opaque Secret with at minimum SECRET_KEY_BASE (64-byte Phoenix session key, mix phx.gen.secret) and DATABASE_URL (or discrete DB_PASSWORD + construct URL at runtime).
• Namespace: guildhall (new).
• DNS record: guildhall.guildhouse.dev → Hetzner LB IPv4 (via Cloudflare). Can be created after LB is provisioned, once the LB IP is known.

Likely shape of the deployment

Based on the keycloak/forgejo pattern:

Namespace:  guildhall
├── Deployment: guildhall-postgres (postgres:16, env POSTGRES_* from guildhall-secrets)
├── PVC:        guildhall-db (longhorn, 5-10Gi)
├── Service:    guildhall-postgres (ClusterIP, 5432)
├── Secret:     guildhall-secrets (SECRET_KEY_BASE, DB_PASSWORD)
├── Deployment: guildhall (image from ghcr / forgejo registry / etc, envs DATABASE_URL + SECRET_KEY_BASE + PHX_HOST=guildhall.guildhouse.dev + PHX_SERVER=true + PORT=4000)
└── Service:    guildhall (type=LoadBalancer, Hetzner annotations, port 80 → 4000)

Release build discipline:

• mix release in Docker multi-stage build (Elixir 1.17.3 / OTP 27 builder stage, debian-slim runtime stage)
• mix ecto.migrate on container start (or a Job, or mix release custom step)
• PHX_SERVER=true to start the HTTP server (per config/runtime.exs)
• Health check endpoint (Phoenix default or custom /health)

Surprises

What's present that wasn't expected:

• Keycloak is already serving at auth.guildhouse.dev. The OIDC substrate Guildhall will eventually integrate with is live. Zero setup needed for that dependency when the time comes.
• cert-manager is installed but unused. Suggests a deliberate deferral in favor of Cloudflare-edge TLS; the ClusterIssuers are staged and ready for when in-cluster TLS is adopted.
• Cilium Envoy DaemonSet is running on every node but with no Gateway API / CiliumEnvoyConfig / L7 policies currently in play. Present for future L7 use, not actively load-bearing yet.

What's expected but absent:

• No HAProxy. Previous K3s-era cluster used HAProxy as ingress; this cluster doesn't. Hetzner LBs took its role.
• v1 substrate foundation is entirely absent from the running cluster. bascule, substrate-operator, chronicle, quartermaster, SPIRE — none running. Flux manifests exist (in the guildhouse-deploy-talos-mirror repo) but are blocked on a dependency chain rooted at missing spire-system namespace and a YAML decode error in cluster-infra/10-cilium-values.yaml. Unblocking this is real work that is NOT on the Red Hat path — governance integration is follow-up.
• No existing Elixir/Phoenix deployment to copy. Guildhall will be the first Phoenix app on this cluster.
• Flux source is on GitHub (guildhouse-deploy-talos-mirror), not Forgejo. Follows the same pattern as the substrate-project umbrella migration just completed — another GitHub→Forgejo item on the cleanup list, not blocking.

Minimum path to Guildhall running at guildhall.guildhouse.dev

1. Dockerfile in ~/projects/substrate-project/guildhall/ — multi-stage with OTP 27, mix release
2. Build and push image to a registry (Forgejo container registry at git.guildhouse.dev/tking/guildhall:v0.1.0 recommended for consistency)
3. Generate SECRET_KEY_BASE via mix phx.gen.secret
4. Create guildhall namespace; create guildhall-secrets Secret
5. Apply Deployment + Service + Postgres + PVC manifest (template from keycloak)
6. Wait for Hetzner LB to provision; note IPv4
7. Create Cloudflare DNS record guildhall.guildhouse.dev → LB IPv4 (proxied, so Cloudflare handles TLS)
8. Verify; run any first-time ecto migration

No cluster infrastructure changes. No cert-manager Certificates. No Flux reconfiguration. No governance-stack dependency. Just the same Deployment-shaped pattern that Keycloak and Forgejo already use, applied to Guildhall.

Governance integration (CRD watchers, SPIFFE identity, Chronicle wiring, Accord enforcement) is explicitly follow-up work for after Guildhall is reachable and the Red Hat submission is in.