guildhall/FORGEJO-REGISTRY-INVESTIGATION-2026-04-21.md

# Forgejo container registry — pre-enablement investigation

**Date:** 2026-04-21
**Scope:** Read-only audit of Forgejo's running state + registry configuration to determine what enablement work (if any) is needed before Guildhall's image push.
**Method:** `kubectl` + `curl` against `https://git.guildhouse.dev`. No mutations.
**Headline:** **The container registry is already enabled.** `/v2/` returns a standard OCI 401, storage headroom is ample (19.4 GB free on 20 GB PVC), and no Forgejo config change is required. Enablement work collapses to credential setup + `docker push`. Estimated time to operational registry for Guildhall: **~30 minutes.**

---

## 1. Forgejo deployment details

| | |
|---|---|
| Namespace | `forgejo` |
| Workload | `Deployment/forgejo` (1 replica, Running) |
| Image | `codeberg.org/forgejo/forgejo:9` |
| Running version | **9.0.3 (Gitea 1.22.0 base)** — confirmed via `GET /api/v1/version` |
| Scheduled node | `gsh-cp-01` (control-plane node, workloads permitted) |
| Companion | `Deployment/forgejo-postgres` (`postgres:16`, 1/1 Running) |
| Init container | `init-config` (renders `/data/gitea/conf/app.ini` from ConfigMap) |
| Runner | `Deployment/forgejo-runner` (0/1 — scaled to zero, source of the Flux health-check warning) |

**Volume mounts on the forgejo container:** one PVC, `data: /data` (the root Forgejo data path; Forgejo 9.x uses `/data` internally, not `/var/lib/gitea` as older Gitea installs did).

**PVCs in the namespace:**

| PVC | Size | StorageClass | Mount |
|---|---|---|---|
| `forgejo-data` | 20 Gi | longhorn | `/data` on forgejo |
| `forgejo-db` | 10 Gi | longhorn | Postgres data |
| `runner-cache` | 5 Gi | longhorn | forgejo-runner (scaled to zero) |

## 2. Forgejo version and config state

### Version

`GET https://git.guildhouse.dev/api/v1/version` → `{"version":"9.0.3+gitea-1.22.0"}`

Forgejo 9.0.3 is a recent release. Container registry / OCI Distribution API support has been GA in Forgejo since the project forked from Gitea (Gitea 1.17+); this version fully supports the container package type.

### Configuration

`forgejo-config` ConfigMap contains the full `app.ini` (40 lines, managed by Flux at path `./k8s/forgejo` in the `guildhouse-deploy-talos-mirror` source repo). Notable sections:

- `[server]` — `DOMAIN=git.guildhouse.dev`, `ROOT_URL=https://git.guildhouse.dev/`, `HTTP_PORT=3000`, `SSH_PORT=22`, `SSH_LISTEN_PORT=2222`, `LFS_START_SERVER=true`
- `[service]` — `DISABLE_REGISTRATION=true` (invite-only signup)
- `[lfs]` — `STORAGE_TYPE=local`
- `[repository]`, `[actions]` — with an `ENABLED = true` that belongs to Actions, not Packages
- **No explicit `[packages]` section.** This is normal for Forgejo 9.x because packages (including container registry) are enabled by default without requiring config-level opt-in.

### Verification that container registry is live

The decisive probe is the OCI Distribution API endpoint root:

```
$ curl -sS -w '%{http_code}\n' https://git.guildhouse.dev/v2/
{"errors":[{"code":"UNAUTHORIZED","message":""}]}
401
```

This is **a standards-compliant OCI registry response** to an unauthenticated request. If the registry were disabled, Forgejo would serve 404 (the endpoint would not be registered). The 401 with a well-formed `errors` object means the registry is routing correctly and simply requires authentication — the default and correct behavior.

Equivalent probe against `/v2/_catalog` returns the same 401 shape.

API-layer probe `GET /api/v1/packages/tking` also returns 401 (`token is required`), consistent with packages being enabled but requiring auth.

### Storage backend

No overridden `[packages.storage]` in app.ini, which means packages use the default local filesystem path under the Forgejo data volume: `/data/gitea/packages/` (or similar Forgejo 9.x path). This lives on `forgejo-data` (the Longhorn 20 Gi PVC), same volume as git repositories, LFS objects, and Forgejo's own state.

## 3. How Forgejo is managed

Forgejo is managed by **Flux**. A `Kustomization` `flux-system/forgejo` reconciles the manifests from:

- **Source:** `GitRepository/flux-system/guildhouse-deploy`
- **URL:** `https://github.com/gh-tking/guildhouse-deploy-talos-mirror`
- **Branch:** `main`
- **Path:** `./k8s/forgejo`
- **Current revision:** `main@169e077f`
- **Interval:** 1 minute

**Kustomization inventory** (what Flux claims to own in this path):

```
_forgejo__Namespace
forgejo_forgejo-config__ConfigMap        ← this is where app.ini lives
forgejo_runner-config__ConfigMap
forgejo_forgejo-secrets__Secret
forgejo_forgejo-http__Service
forgejo_forgejo-postgres__Service
forgejo_forgejo_apps_Deployment
forgejo_forgejo-postgres_apps_Deployment
forgejo_forgejo-runner_apps_Deployment
forgejo_forgejo-data__PersistentVolumeClaim
forgejo_forgejo-db__PersistentVolumeClaim
```

**Status:** `Ready: False` / `Healthy: False` because of a health-check timeout on `forgejo-runner` — but this is a scaled-to-zero sidecar Deployment, not a problem with core Forgejo. The core Forgejo Deployment is Ready, the registry is live, and the Kustomization IS reconciling successfully against new commits — the health condition is just stuck on the runner.

**Consequence:** if we ever needed to change Forgejo's `app.ini` (we don't, for registry work), the mechanism is to edit `k8s/forgejo/forgejo-config.yaml` in the `gh-tking/guildhouse-deploy-talos-mirror` GitHub repo, push to `main`, and wait for Flux to reconcile (1-minute interval). This path is functional today despite the runner health warning.

## 4. The cluster-infra Flux error

`kubectl describe kustomization cluster-infra -n flux-system`:

- **Suspend: true** (explicitly suspended by an operator earlier)
- **Source:** `guildhouse-deploy` GitRepository, path `./talos/manifests/cluster-infra`
- **Error message:**

```
failed to decode Kubernetes YAML from /tmp/kustomization-.../talos/manifests/cluster-infra/
10-cilium-values.yaml: missing Resource metadata <nil>
```

**Diagnosis:** `10-cilium-values.yaml` is a Helm values file being handed to kustomize-controller as if it were a raw Kubernetes manifest. The file doesn't have a `kind` or `metadata` — it's a values document intended to be consumed by `helm install --values`, not a standalone Kubernetes resource. Kustomize chokes because every file in a Kustomization source path is expected to be Resource-shaped.

**Fix severity:** trivial. One of:
- Move `10-cilium-values.yaml` into a `values/` subdirectory that isn't referenced by `kustomization.yaml`
- Rename the file so it doesn't get picked up (e.g., `10-cilium-values.yaml.hold`)
- Add a `kustomization.yaml` with explicit `resources:` that excludes it
- Replace the file with a proper `HelmRelease` CR that references the values externally

Any of these is a single-file source edit, Flux reconciles on next push.

**Time estimate:** ~30–60 minutes including the commit+push+reconcile+verify cycle. The main complication is that `cluster-infra` has `Suspend: true` — whoever suspended it did so deliberately (likely because the error was cascading to blocked downstream Kustomizations). Un-suspending should probably wait until the underlying YAML is fixed, otherwise the same error re-appears.

**Crucially: this error does NOT block Forgejo registry work or Guildhall deployment.** The two Kustomizations are independent. Guildhall deployment can proceed entirely outside the Flux chain (direct `kubectl apply` or a new Guildhall-specific Kustomization once registry+deploy are working). The cluster-infra/spire/quartermaster/bascule chain is substrate-foundation work that's explicitly follow-up.

## 5. Cluster image pull pattern

**No existing pattern for private-registry pulls.** The entire cluster currently pulls only from public registries:
- `quay.io/keycloak/keycloak:26.0`
- `codeberg.org/forgejo/forgejo:9`
- `postgres:16` (Docker Hub)
- `quay.io/cilium/cilium:v1.16.5` and `quay.io/cilium/cilium-envoy`
- Longhorn and Flux images (all public)

Specifically:

```
$ kubectl get secrets -A --field-selector type=kubernetes.io/dockerconfigjson
No resources found
```

Zero `dockerconfigjson` secrets cluster-wide. Zero `imagePullSecrets` referenced on any Deployment.

**Guildhall will be the first workload pulling from a private Forgejo registry.** It introduces the pattern, which then becomes the template for subsequent workloads. Two options:

1. **Make the `tking/guildhall` Forgejo package public.** Forgejo packages can be scoped public or private; a public container package allows anonymous pulls and no pull secret is needed. This matches the rest of the cluster's zero-pull-secret state. Appropriate if there's nothing sensitive in the image itself.
2. **Keep the package private and add a `dockerconfigjson` Secret.** Standard pattern: `kubectl create secret docker-registry guildhall-registry --docker-server=git.guildhouse.dev --docker-username=<user> --docker-password=<token>`, then reference in the Deployment via `imagePullSecrets: [name: guildhall-registry]`.

Option 1 is simplest for v0.1. Option 2 is better hygiene long-term.

## 6. Storage headroom on Forgejo's volume

`kubectl exec -n forgejo deployment/forgejo -- df -h` (inside the forgejo container):

```
/dev/longhorn/pvc-683ec33a-...    19.5G    137.2M    19.4G   1%   /data
```

**Headroom is ample.** 19.4 GB free on a 20 GB PVC. Current Forgejo usage after 10 days is 137 MB (git repos + LFS + internal state).

A Guildhall container image — Elixir release on debian-slim, typically 100-300 MB compressed per tag, with OCI layer deduplication across tags — would add maybe 1-3 GB of package storage over dozens of iterations. No pressure on the volume for the foreseeable future.

**No resize required.** If long-term registry growth becomes an issue (multiple applications all pushing many tags, or large binary releases), Longhorn supports online expansion of the PVC — but that's a much-later concern.

---

## Synthesis

### Is the registry already enabled?

**Yes.** The `/v2/` and `/v2/_catalog` endpoints return proper OCI Distribution API responses (401 unauthenticated with well-formed `errors` objects). Forgejo 9.x enables packages by default; no `[packages]` config section is needed, and none is present. The registry is live and waiting for an authenticated client.

### What enablement work is required?

**None at the Forgejo-config layer.** The only work is client-side:

1. **Create a Forgejo Personal Access Token** (scope: `package:write`) via the Forgejo UI at `https://git.guildhouse.dev/-/user/settings/applications`
2. **Docker login from the build machine:** `docker login git.guildhouse.dev -u tking -p <PAT>`
3. **Build + push** the Guildhall image: `docker build -t git.guildhouse.dev/tking/guildhall:v0.1.0 . && docker push …`
4. **Set package visibility** in Forgejo — public (anon-pull, no imagePullSecret needed) or private (create a `dockerconfigjson` Secret in the `guildhall` namespace, reference in Deployment)

No Flux source edits. No Kustomization changes. No ConfigMap changes. No `cluster-infra` unblock required.

### Is the `cluster-infra` Flux error a blocker?

**No.** The Forgejo registry operates entirely outside the cluster-infra / spire / quartermaster / bascule Flux chain. Forgejo is managed by its own independent Kustomization (`flux-system/forgejo`), which is successfully reconciling against source revisions even though its Ready condition is flagged False by the unrelated forgejo-runner health check.

The `cluster-infra` error is real and worth fixing separately (trivial single-file fix in the GitHub source repo) but it has zero coupling to registry enablement or Guildhall deployment. Treat as a cleanup backlog item, not a pre-req.

### Estimated time to registry operational

| Step | Time |
|---|---|
| Create Forgejo PAT (Forgejo UI) | 2 min |
| `docker login git.guildhouse.dev` | <1 min |
| Dockerfile + `mix release` setup in Guildhall repo | 15-20 min (real work) |
| `docker build` (cold build for Elixir + OTP + mix deps + assets) | 5-10 min |
| `docker push` | 1-3 min (single tag, ~200 MB compressed) |
| Set package visibility (public or private + pull secret) | 2-5 min |
| **Total to first successful image in the registry** | **~30-45 min** |

Most of the time is the Dockerfile + release-build setup, not the registry interaction itself.

### Recommended next step

**Build the Guildhall Dockerfile and push a first image.** Sequencing:

1. Author `Dockerfile` in `~/projects/substrate-project/guildhall/` — multi-stage (Elixir 1.17.3/OTP 27 builder → debian-slim runtime, `mix release`, non-root user, expose 4000, healthcheck endpoint)
2. Author `.dockerignore` that excludes `_build/`, `deps/`, `.git/`, `priv/static/` (if built separately) — matches Phoenix release conventions
3. Create Forgejo PAT with `package:write` scope
4. `docker login git.guildhouse.dev` from the desktop
5. `docker build -t git.guildhouse.dev/tking/guildhall:v0.1.0 .`
6. `docker push git.guildhouse.dev/tking/guildhall:v0.1.0`
7. Verify via Forgejo UI at `https://git.guildhouse.dev/tking/-/packages/container/guildhall` and via `curl` to `/v2/tking/guildhall/manifests/v0.1.0` (authenticated)
8. Decide package visibility, and if private, create `guildhall-registry` Secret in the `guildhall` namespace (namespace doesn't exist yet — create at deploy time)

The Kubernetes-side deploy (Deployment + Service + Postgres + PVC + Secret) proceeds in parallel with or immediately after the image build, following the Keycloak pattern captured in the earlier `DEPLOY-EXPLORATORY-2026-04-21.md`.

No pre-work needed on Forgejo itself. The registry is ready.