kedge/docs/hfl-kedge-integration.md

HFL–Kedge Network Integration

How Host Function Layer WASM module Shellstream sessions interact with Kedge's CNI plugin and DaemonSet.

Cross-references: Substrate docs/shell-primitive.md — BPF map structures, eBPF hooks, shell lifecycle. Substrate docs/hfl-spec.md — WASM module grant attenuation, host function categories. Kedge internal/cni/plugin.go — CNI CmdAdd flow. Kedge internal/cni/policy.go — SVID-scoped policy evaluation. Kedge internal/quartermaster/session_transit.go — Session transit artifact type.

---

1. Overview

Kedge is the network data plane for the Guildhouse infrastructure. It operates as both a Kubernetes CNI plugin (creating the net1 secondary interface via Multus) and a DaemonSet managing WireGuard tunnels, VLAN interfaces, and Shellstream session termination. The Host Function Layer (HFL) is a WASM-based runtime embedded in Bascule that exposes structured host functions — Quartermaster queries, telemetry reads, infrastructure state, YANG operations — to application workloads running inside attested shells.

When an HFL WASM module calls a host function that requires network access, the resulting Shellstream session traverses Kedge's network stack. The HFL does not interact with Kedge directly — it opens Shellstream connections that transit Kedge's net1 interface like any other pod traffic. What makes HFL sessions distinct is the three-layer enforcement model that governs them:

1. Kernel layer (eBPF shell): The shell_map BPF map restricts which destinations and modes the pod's cgroup can reach. Kedge's CNI plugin reads allowed_mode from this map at interface creation time. The eBPF CGROUP_SOCK_ADDR hook enforces allowed_targets at connection time. These checks happen in the kernel — no userspace daemon is in the hot path.

2. Application layer (HFL grants): Before a host function opens any Shellstream session, the HFL runtime validates the WASM module's ModuleGrant against the requested operation. If the grant does not cover the operation (wrong capabilities, wrong registry, wrong mode), the request is rejected before it reaches the network.

3. Cryptographic layer (SAT attenuation): The per-request RequestToken carried in the Shellstream frame header is validated independently by the receiving service (Quartermaster, Bascule, Prometheus). Each layer operates without trusting the others.

These three layers compose but do not depend on each other. A compromised HFL runtime cannot bypass the eBPF shell. A compromised eBPF program cannot forge SAT tokens. This is defense in depth through independent enforcement points, not redundant checks in a single trust domain.

---

2. Shared BPF Map Architecture

Kedge and the shell primitive share state through shell_map, a BPF_MAP_TYPE_HASH map keyed by cgroup_id. There is no API intermediary, no sidecar, no IPC channel. Both the eBPF enforcement programs and Kedge's CNI plugin read from the same map.

Map definition

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __type(key, __u64);                 /* cgroup_id */
    __type(value, struct shell_state);
    __uint(max_entries, 1024);
} shell_map SEC(".maps");

shell_state layout (212 bytes)

struct shell_state {
    __u8  sat_hash[32];                         /* SHA-256 of the bound SAT */
    __u32 capabilities;                         /* Capability bitmask */
    __u32 allowed_mode;                         /* Overlay/underlay mode bitmask */
    __u32 num_allowed_targets;                  /* Count of populated entries */
    struct shell_target allowed_targets[16];    /* Destination whitelist */
    __u64 accord_id;                            /* Hash of the governing accord */
    __u64 session_counter;                      /* Monotonic counter for correlation IDs */
    __u64 created_at;                           /* Nanosecond timestamp (CLOCK_MONOTONIC) */
    __u64 last_updated;                         /* Nanosecond timestamp (CLOCK_MONOTONIC) */
    __u32 flags;                                /* Shell state flags */
    __u32 _padding;                             /* Alignment to 8-byte boundary */
};

struct shell_target {
    __be32 addr;        /* IPv4 destination (network byte order) */
    __be16 port;        /* Destination port (0 = any port) */
    __u8   protocol;    /* IPPROTO_TCP (6), IPPROTO_UDP (17), 0 = any */
    __u8   flags;       /* Per-target flags */
};

Fields relevant to Kedge

Field	Kedge usage
allowed_mode	CNI plugin reads this at CmdAdd time to decide which route types to program on net1. OVERLAY (0x1) → WireGuard routes. UNDERLAY (0x2) → VLAN bridge routes. Both bits → both route sets.
allowed_targets	eBPF CGROUP_SOCK_ADDR hook validates outbound connections against this whitelist at connect time. Kedge does not enforce targets — the kernel does.
capabilities	READ (0x1), PROPOSE (0x2), MUTATE (0x4), ADMIN (0x8). Kedge uses this when building SessionTransitArtifact records. The DaemonSet's health endpoint exposes capability distribution as Prometheus metrics.
flags	ACTIVE (0x1), FROZEN (0x2), DRAINING (0x4). DaemonSet watches flag transitions. When a shell enters FROZEN, the DaemonSet stops accepting new Shellstream handshakes for that session. When DRAINING, it tears down associated tunnel state.
sat_hash	Included in every SessionTransitArtifact for Quartermaster governance chain linkage.
accord_id	Used by the DaemonSet to look up the local accord policy that governs capability grants for incoming Shellstream sessions.

Access pattern

The BPF map is pinned at /sys/fs/bpf/shell_map. Kedge accesses it from Go via cilium/ebpf map operations:

• CNI plugin (CmdAdd): Looks up the pod's cgroup_id, reads allowed_mode and flags. This is a single MapLookupElem call — O(1), no syscall overhead beyond the BPF command. If no entry exists (shell is UNBOUND), the CNI plugin installs no routes on net1, effectively isolating the pod.
• DaemonSet: Subscribes to shell_map changes via the shell_events ring buffer (see ShellProgrammer.subscribe_events() in shell-primitive.md §7.1). Reacts to bind_shell, freeze_shell, drain_shell, and destroy_shell events.

---

3. Routing HFL Shellstream Sessions Through Kedge

An HFL-originated Shellstream session follows the same packet path as any other pod egress through net1. The difference is that three independent checks have already passed before the first SYN reaches the wire. This section traces the four common HFL session types.

3.1 Overlay: HFL → Quartermaster (cross-cluster query)

A WASM module calls query-records on the session-transit registry hosted by a remote Quartermaster instance.

WASM module
  │ host.quartermaster.query_records("session-transit", filter, limit)
  ▼
HFL Runtime
  │ 1. Validate ModuleGrant: capabilities includes READ,
  │    registries includes "session-transit",
  │    allowed_mode includes OVERLAY
  │ 2. Construct RequestToken (single_use=true, expires=30s)
  │ 3. Open TCP connection to Quartermaster endpoint
  ▼
eBPF CGROUP_SOCK_ADDR hook
  │ 4. Look up cgroup_id in shell_map
  │ 5. Check flags: ACTIVE (not FROZEN/DRAINING)
  │ 6. Check destination against allowed_targets[0..N]
  │ 7. Permit or deny at kernel level
  ▼
net1 interface (Kedge CNI)
  │ 8. Packet hits overlay route → forwarded to wg0
  ▼
WireGuard tunnel (wg0)
  │ 9. Encrypted transit to remote cluster
  ▼
Remote Kedge DaemonSet (Shellstream listener)
  │ 10. 3-way handshake: ATTEST-INIT / ATTEST-VERIFY / ATTEST-CONFIRM
  │ 11. Validate SAT, evaluate capability against local accord
  │ 12. Record SessionTransitArtifact
  ▼
Quartermaster
  │ 13. Validate RequestToken independently
  │ 14. Execute query, return results

Steps 1–2 are HFL enforcement (application layer). Steps 4–7 are eBPF enforcement (kernel layer). Step 8 depends on Kedge having programmed overlay routes at CmdAdd time — which only happens if shell_state.allowed_mode & OVERLAY != 0. Steps 10–13 are cryptographic enforcement (SAT + RequestToken).

3.2 Underlay: HFL → Bascule SDK dispatch (device read)

A WASM module calls get-device-state for a FortiGate on the local VLAN.

WASM module
  │ host.infrastructure.get_device_state("fortigate.transit.local")
  ▼
HFL Runtime
  │ 1. Validate ModuleGrant: capabilities includes READ,
  │    allowed_mode includes UNDERLAY
  │ 2. Construct RequestToken
  │ 3. Open Shellstream to Bascule (may be local or cross-cluster)
  ▼
eBPF CGROUP_SOCK_ADDR hook
  │ 4. Validate against shell_map (same as §3.1)
  ▼
net1 interface (Kedge CNI)
  │ 5. Packet hits underlay route → forwarded via VLAN bridge
  ▼
VLAN bridge (br-mgmt / tagged interface)
  │ 6. Frame reaches managed device on infrastructure VLAN
  ▼
Bascule SDK dispatch
  │ 7. Validate RequestToken
  │ 8. Execute read-only SDK call (e.g., fortiosapi.get)
  │ 9. Return device state to WASM module

The underlay path differs at step 5: traffic exits through the VLAN bridge rather than the WireGuard tunnel. The CNI plugin only programs underlay routes if shell_state.allowed_mode & UNDERLAY != 0.

3.3 Local: HFL → YANG compiler (no Shellstream)

YANG host functions (validate-instance, compile-dry-run, diff-state) call the Python YANG compiler within the same container. No Shellstream session is opened and no traffic traverses net1. The HFL runtime invokes the compiler as a local subprocess. Kedge is not involved.

This is worth documenting because it is the exception: HFL host functions do not always produce network traffic. The YANG category is purely local computation.

3.4 Local: HFL → Prometheus (telemetry query)

Telemetry queries (query-prometheus, query-timescale) may target a Prometheus instance on the same cluster. When the destination is local (same node or same cluster overlay), the path is:

WASM module → HFL Runtime → eBPF check → net1 → overlay route → wg0 → local Prometheus

Even local-cluster queries transit net1 and the WireGuard tunnel because Kedge treats all overlay traffic uniformly. There is no "local bypass" optimization — this simplifies the enforcement model by ensuring every session, regardless of locality, passes through the same BPF map checks.

---

4. CNI Plugin: shell_state-Aware Route Programming

The CNI plugin's CmdAdd handler programs routes on the pod's net1 interface based on shell_state.allowed_mode. This is the critical integration point: Kedge decides at pod creation time what network paths are available.

Current CmdAdd flow (from internal/cni/plugin.go)

ParseNetConf → loadMeshTopology → createVethPair → attachOverlayRoutes
  → attachUnderlayRoutes → programPodRoutes → applySVIDPolicy

Extended flow with shell_state integration

ParseNetConf → loadMeshTopology → createVethPair
  → lookupShellState(cgroup_id)
  → if shell_state is UNBOUND: return (no routes, pod is isolated)
  → if flags & ACTIVE == 0: return (shell not active, pod is isolated)
  → if allowed_mode & OVERLAY: attachOverlayRoutes
  → if allowed_mode & UNDERLAY: attachUnderlayRoutes
  → programPodRoutes (only for modes granted above)
  → applySVIDPolicy (SVID check is orthogonal to shell_state)

lookupShellState pseudocode

func lookupShellState(cgroupID uint64) (*ShellState, error) {
    m, err := ebpf.LoadPinnedMap("/sys/fs/bpf/shell_map", nil)
    if err != nil {
        return nil, fmt.Errorf("shell_map not available: %w", err)
    }
    defer m.Close()

    var state ShellState
    if err := m.Lookup(cgroupID, &state); err != nil {
        // No entry = UNBOUND. Fail closed: no routes.
        return nil, nil
    }
    return &state, nil
}

Route programming rules

allowed_mode	Overlay routes	Underlay routes	Effect
0x0 (none)	No	No	Pod is network-isolated on net1.
0x1 (OVERLAY)	Yes	No	Pod can reach remote clusters via WireGuard. Cannot reach infrastructure VLANs.
0x2 (UNDERLAY)	No	Yes	Pod can reach infrastructure devices on local VLANs. Cannot reach remote clusters.
0x3 (both)	Yes	Yes	Full access. Typical for admin-tier shells.

Fail-closed behavior

If shell_map is not pinned (e.g., the shell primitive is not yet loaded), the CNI plugin treats this as equivalent to UNBOUND — no routes are programmed. This is intentional: a pod without an active shell binding has no business reaching Kedge-managed networks.

If shell_map exists but contains no entry for the pod's cgroup_id, the same fail-closed behavior applies. The CNI plugin does not fall back to permissive routing.

---

5. DaemonSet: shell_state Watch and Lifecycle Coordination

The DaemonSet reacts to shell lifecycle transitions by adjusting tunnel state, Shellstream session handling, and telemetry.

Event subscription

The DaemonSet calls ShellProgrammer.subscribe_events() (implemented via the shell_events BPF ring buffer) to receive shell state transitions. In Go, this is accessed through the cilium/ebpf ring buffer reader:

reader, err := ringbuf.NewReader(shellEventsMap)
for {
    record, err := reader.Read()
    event := parseShellEvent(record.RawSample)
    switch event.Type {
    case EventBindShell:    handleBind(event)
    case EventFreezeShell:  handleFreeze(event)
    case EventDrainShell:   handleDrain(event)
    case EventDestroyShell: handleDestroy(event)
    }
}

Lifecycle event handling

Event	DaemonSet action
bind_shell	Register the shell's cgroup_id in local session tracking. If the shell has allowed_mode & OVERLAY, ensure the associated WireGuard peers are configured. Log the binding with sat_hash and accord_id.
freeze_shell	Stop accepting new Shellstream handshakes for sessions associated with this cgroup_id. Existing established sessions continue (the shell is being rotated, not destroyed). Increment kedge_shell_freeze_total counter.
drain_shell	Stop accepting new handshakes AND begin tearing down established Shellstream sessions for this cgroup_id. The mesh manager marks associated peers as draining. Increment kedge_shell_drain_total counter.
destroy_shell	Remove all local state for this cgroup_id. Clean up any tunnel state that was exclusively serving this shell. Emit final telemetry. Increment kedge_shell_destroy_total counter.

SAT rotation coordination

When Bascule rotates a SAT, it freezes the shell, updates sat_hash, then unfreezes. The DaemonSet sees two events: freeze_shell followed by an update_shell (which carries the new sat_hash), then an implicit unfreeze (the FROZEN flag is cleared). During the freeze window, no new Shellstream handshakes are accepted, but existing sessions continue. This prevents a window where the old SAT is invalid but the new one hasn't propagated — the freeze ensures atomicity from the network's perspective.

---

6. Quartermaster Records for HFL Sessions

HFL-originated sessions produce the same SessionTransitArtifact and NetworkMutationArtifact records as any other Shellstream session. However, HFL sessions carry additional provenance that should be captured for auditability.

Extended SessionTransitArtifact

The existing artifact (from internal/quartermaster/session_transit.go) is extended with optional HFL-specific fields:

type SessionTransitArtifact struct {
    // ... existing fields (session_id, sat_hash, source/dest cluster, etc.)

    // HFL provenance (optional — zero values when session is not HFL-originated).
    HFLModuleName      string `json:"hfl_module_name,omitempty"`
    HFLFunctionName    string `json:"hfl_function_name,omitempty"`
    HFLGrantHash       []byte `json:"hfl_grant_hash,omitempty"`
    HFLRequestTokenHash []byte `json:"hfl_request_token_hash,omitempty"`
}

Field	Source	Purpose
hfl_module_name	Shellstream frame header, set by HFL runtime	Identifies which WASM module initiated the session (e.g., "telemetry-query").
hfl_function_name	Shellstream frame header, set by HFL runtime	Identifies which host function was called (e.g., "query_flow_records").
hfl_grant_hash	SHA-256 of the ModuleGrant	Links the session back to the specific grant that authorized it. Enables audit queries like "show all sessions authorized by grant X".
hfl_request_token_hash	SHA-256 of the RequestToken	Links the session to the single-use per-request token. Since tokens are single-use, this provides a 1:1 mapping from token to session.

These fields are populated by the Kedge DaemonSet's Shellstream listener when the ATTEST-INIT message includes HFL provenance headers. Non-HFL sessions (direct Shellstream connections) leave these fields empty.

NetworkMutationArtifact — no HFL extension needed

Network mutations are dispatched through Bascule SDK, not through HFL host functions. The HFL can propose mutations (via submit-proposal with PROPOSE capability), but the actual execution goes through Bascule's ceremony workflow. By the time Kedge records a NetworkMutationArtifact, the mutation has already been approved through a ceremony and dispatched by Bascule — the HFL origin is captured in the ceremony's proposal chain, not in the mutation artifact itself.

Canonical serialization

The CanonicalBytes() method on SessionTransitArtifact must include the HFL fields when present. The RFC 8785 (JCS) canonical form ensures deterministic hashing regardless of field ordering. Fields with zero values (omitempty) are excluded from the canonical form to maintain backward compatibility with non-HFL session records.

---

7. Mode Authorization: End-to-End Flow

This section traces the complete authorization chain for an HFL-originated underlay session, showing how the three enforcement layers compose.

Scenario

A WASM module "proposal-builder" calls compile-dry-run (local, no network — see §3.3), then calls submit-proposal to Quartermaster (overlay), then an operator approves the ceremony and Bascule dispatches the mutation (underlay). We trace the overlay leg (proposal submission) end to end.

Step 1: SAT issuance (before Kedge)

The Vigil ceremony issues a SAT with:

• capabilities: READ | PROPOSE (0x3)
• allowed_mode: OVERLAY (0x1) — this module does not need underlay access
• scope: registries ["network-mutation", "session-transit"]

Bascule calls bind_shell(cgroup_id, shell_state) with these values. The shell_map entry is created.

Step 2: Grant attenuation (HFL)

The HFL runtime derives a ModuleGrant for "proposal-builder":

ModuleGrant {
    module_hash:    sha256("proposal-builder.wasm"),
    parent_sat_hash: shell_state.sat_hash,
    capabilities:   PROPOSE (0x2),        // attenuated from READ|PROPOSE
    registries:     ["network-mutation"],  // attenuated from full list
    allowed_mode:   OVERLAY (0x1),        // same as SAT (cannot widen)
    expires_at:     min(sat.expires, now + 1h),
}

Attenuation is strictly narrowing: grant.capabilities & sat.capabilities == grant.capabilities. The grant cannot add MUTATE or UNDERLAY that the SAT does not carry.

Step 3: Request token (HFL, per-call)

When the module calls submit-proposal, the HFL constructs:

RequestToken {
    grant_hash:     sha256(ModuleGrant),
    operation:      "submit-proposal",
    parameters_hash: sha256(serialized_proposal),
    single_use:     true,
    expires_at:     now + 30s,
}

Step 4: eBPF enforcement (kernel)

The connect() syscall triggers the CGROUP_SOCK_ADDR hook:

1. Look up cgroup_id in shell_map → finds the shell_state.
2. Check flags: ACTIVE is set, FROZEN is not → proceed.
3. Check destination IP/port against allowed_targets[0..num_allowed_targets] → match found.
4. Permit the connection.

If the destination were on an underlay VLAN (and allowed_mode only has OVERLAY), this check would still pass at the eBPF level — allowed_targets is an explicit whitelist, not mode-based. The mode enforcement happens at the routing level (step 5).

Step 5: Kedge route enforcement (CNI)

The packet exits the pod via net1. Because the CNI plugin read allowed_mode = OVERLAY at CmdAdd time, only overlay (WireGuard) routes are programmed. The destination Quartermaster endpoint resolves to an overlay route through wg0. If the destination were on an underlay VLAN, there would be no route — the packet would be dropped by the kernel's routing table. This is Kedge's enforcement: not a firewall rule, but the absence of a route.

Step 6: WireGuard transit (Kedge mesh)

The packet traverses the WireGuard tunnel to the remote cluster. Kedge's mesh manager maintains the peer configuration and monitors tunnel health.

Step 7: Remote Shellstream handshake (Kedge DaemonSet)

The remote Kedge DaemonSet receives the connection on its Shellstream listener:

1. Read ATTEST-INIT (includes SAT token, HFL provenance headers).
2. Validate SAT against local SPIRE trust bundle.
3. Evaluate capability request against local accord policy (see internal/shellstream/capability.go).
4. Send ATTEST-VERIFY with granted capabilities.
5. Read ATTEST-CONFIRM.
6. Record SessionTransitArtifact with HFL fields populated.

Step 8: Quartermaster validation (independent)

Quartermaster receives the submit-proposal RPC with the RequestToken in metadata:

1. Verify RequestToken.signature against the HFL runtime's signing key.
2. Check single_use — mark token as consumed.
3. Check expires_at — reject if expired.
4. Verify grant_hash chains back to a valid SAT.
5. Check that the grant's capabilities include PROPOSE.
6. Accept the proposal for ceremony processing.

Enforcement summary

Layer	What it checks	Where it runs	Failure mode
HFL grant	Module has appropriate capability and registry scope	Userspace (HFL runtime, same pod)	Request rejected before network
eBPF shell	Destination is in allowed_targets, shell is ACTIVE	Kernel (CGROUP_SOCK_ADDR hook)	connect() returns EACCES
Kedge routes	allowed_mode permits the network path type	Kernel (routing table, programmed by CNI)	Packet dropped (no route to host)
Shellstream handshake	SAT is valid, capabilities match local accord	Userspace (remote Kedge DaemonSet)	Handshake rejected at ATTEST-VERIFY
Service token	RequestToken is valid, single-use, unexpired	Userspace (Quartermaster/Bascule)	RPC rejected with PermissionDenied

Five independent checks, three distinct trust domains (pod kernel, pod userspace, remote service), two cryptographic validations (SAT, RequestToken). No single point of compromise grants unauthorized access.