Kubernetes executor boundary
The runnable lab proves an out-of-process, typed, mutually authenticated executor boundary. Its default backend remains the durable pod simulator: no standard application-lab command reads a Kubernetes context. The separate SOTA profile creates its own cluster and qualifies the real NetworkPolicy adapter behind a separate double opt-in.
Wire contract
The control plane can call exactly three JSON operations over TLS 1.3:
| Endpoint | Input | Result |
|---|---|---|
POST /v1/prepare |
one validated ActionIntent |
fresh state hash and preconditions |
POST /v1/execute |
intent, state hash, signed 15-second grant and ExecutionProof |
before/after state and postconditions |
POST /v1/restore |
durable job plus signed restoration grant | conflict-safe restoration result |
There is no shell, command, manifest, arbitrary URL, generic Kubernetes proxy, generic patch, or caller-selected API verb. Both sides verify a single exact SPIFFE-format URI SAN. mTLS is necessary but insufficient: the executor also verifies an Ed25519 signature, exact audience, intent/state hashes and time window from a pinned public-key bundle. It atomically claims each execution grant ID in PostgreSQL before mutation. The bundle contains no signing secret or OpenBao token. Responses are size-limited and schema-validated again by the client. Transport ambiguity is an execution failure; the signed restoration obligation has already been persisted and scheduled before execution begins.
Real Kubernetes adapter algorithm
KubernetesNetworkPolicyExecutor implements this fixed sequence behind the same
ActionExecutor interface:
- Resolve an allow-listed cluster alias; never accept a kubeconfig path or API address from a request.
GETthe exact pod, UID, controller owner andresourceVersion, then list the namespace and require at least two ready siblings owned by the same controller UID.- Create an AEGIS-owned deny-all NetworkPolicy selecting a deterministic
quarantine id, recheck the durable emergency stop, then add only the reserved
pod label with JSON Patch
testoperations on UID andresourceVersion. - Re-read the pod and policy and return only the typed postconditions.
- At restoration time, require the original UID and exact ownership markers, delete the policy with UID/resourceVersion preconditions, then remove only the exact reserved label with another conditional JSON Patch. Unrelated newer pod fields are never replaced.
The compact numeric resourceVersion in the generic protocol is accompanied by
the exact Kubernetes UID and string resourceVersion; the latter is what every
real mutation tests. NetworkPolicy names and selectors are derived from the
target plus UID, not caller-controlled text.
lab/kubernetes/executor-rbac.yaml is namespace-scoped and intentionally avoids
a ClusterRoleBinding. Kubernetes RBAC cannot constrain which fields a patch
may alter, and resourceNames cannot safely scope dynamic creates.
lab/kubernetes/executor-admission-policy.yaml therefore adds fail-closed
ValidatingAdmissionPolicy guards for Kubernetes 1.30+. Regenerate its explicit
namespace rule for every approved namespace. The checked-in form has been
validated on Kubernetes 1.36.1; every other version must be qualified before
enforcement.
The stricter proof-carrying profile adds a separate admission process that has
public verification keys but no Kubernetes mutation credential. The executor
places the bounded aegis.execution-proof.v1 and its hash in reserved
annotations; the verifier independently re-derives policy semantics, mission
attenuation, state freshness, target UID, restoration obligation and every
signature. For restoration, it authenticates the expired outer proof but
requires the embedded, longer-lived restoration grant to be due and unexpired.
Its trusted configuration pins the cluster alias, execution audience and exact
executor ServiceAccount instead of learning them from the proof. Pod updates
must differ only by the reserved label/proof annotations, NetworkPolicies must
be exact ingress/egress deny-all objects, and restoration cannot swap the proof
that was attached when the pod was quarantined.
lab/kubernetes/proof-admission-webhook.template.yaml is deliberately a
template and is not applied by the local lab. Replace its immutable image digest
and CA bundle, set AEGIS_ADMISSION_EXPECTED_AUDIENCE and
AEGIS_ADMISSION_CLUSTER_ALIAS, provision only the public-key ConfigMap and TLS Secret, deploy in
a disposable cluster, and inspect audit events before enforcing it. A malformed
template must fail deployment rather than silently bypass verification.
lab/kubernetes/executor-networkpolicy.yaml defaults the executor to no egress.
It deliberately omits Kubernetes-API egress because that endpoint is
cluster-specific. Deployment automation must inject a reviewed exact API
destination; the checked-in default remains non-operational and fail-closed.
Explicit real-mode activation
This mode writes to a Kubernetes cluster. It is not used by any smoke test and must be enabled only for a disposable, explicitly authorized cluster:
$env:AEGIS_KUBERNETES_MODE = 'real'
$env:AEGIS_REAL_KUBERNETES_CONFIRM = 'quarantine-networkpolicy-v1'
$env:AEGIS_KUBECONFIG = 'C:\absolute\reviewed\executor-only.kubeconfig'
$env:AEGIS_KUBERNETES_CONTEXT = 'aegis-disposable'
$env:AEGIS_KUBERNETES_CLUSTER_ALIAS = 'lab-eu-1'
$env:AEGIS_KUBERNETES_ALLOWED_NAMESPACES = 'payment-prod'
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/lab-start.ps1 `
-WithPostgresAuthority -WithTemporal -WithOpenBao -WithRemoteExecutor
The kubeconfig path must be absolute. The loader never calls
loadFromDefault, rejects HTTP, skipped TLS verification, kubeconfig proxies,
basic authentication, impersonation, exec and auth-provider plugins, and
requires embedded token or client-certificate credentials. The target cluster
alias and namespace come from configuration, never from a request-selected API
server.
Do not use an administrator kubeconfig. npm run sota:qualify creates separate
30-minute TokenRequest credentials for the executor and a read-only observer.
It proves both RBAC profiles, lets the observer confirm the quarantine objects
without write permission, proves admission denials, and observes Cilium block
and then restore traffic. For
any other cluster, first apply the Role/RoleBinding and admission policies, add
an exact API-server egress rule, then repeat conflict, stop and restoration
tests.
SPIRE handoff
lab/spire/workload-identities.yaml contains separate workload registrations
for the control plane and executor. The SOTA profile installs Server, Agent,
CSI driver and Controller Manager, then npm run sota:spire:qualify proves that
exactly one executor workload receives its X.509-SVID. The Windows application
lab still uses its local CA; it must not be presented as end-to-end SPIRE until
those services run as attested workloads.
The node cordon executor, and what it costs in RBAC
KubernetesNodeCordonExecutor performs the second contracted action: setting
spec.unschedulable on a node for a bounded window. It follows the same
sequence — read, compare-and-set on UID and resourceVersion, an ownership
label so restoration can prove the change was ours, the execution proof in
reserved annotations — and shares the API-error handling that keeps a target
name out of the logs. It shares no mechanism with the NetworkPolicy path, which
is why it is a separate class rather than a branch.
This is a wider grant than the pod path, and the difference is not a detail.
A node is a cluster-scoped object. Patching one requires a ClusterRole and a
ClusterRoleBinding, so the namespace boundary that bounds the quarantine
executor does not exist here. Three things bound it instead, and all three are
narrower than RBAC can express:
allowedNodeson the executor. A cordon outside it is refused before any read reaches the API server. An empty set means the executor can cordon nothing, which is the correct default for a deployment that has not decided.- The mission's
allowedActions. A mission that permits quarantine does not permit cordoning; the adversarial harness has a route for exactly this. - The safety floor.
preparerefuses unless another node is ready, schedulable and not being deleted — a node that is draining reportsReadyright up to the moment it stops, and counting it as spare capacity would make the floor decorative.
lab/kubernetes/cordon-executor-rbac.yaml is the grant, and it is deliberately
absent from kustomization.yaml so that a kustomize build never applies it by
accident. resourceNames confines patch to named nodes; it cannot confine
list, which the safety floor needs in order to see the whole set. A deployment
that does not need cordoning should not grant the ClusterRole at all. Running only KubernetesNetworkPolicyExecutor is the supported way to do
that: the kernel reads its supportedActions, and a cordon proposal is refused
with action_not_implemented before any authority is issued rather than
failing at execution with a capability already spent.
Running both
RoutingExecutor puts several executors behind one door, choosing by action
type. The property it exists to preserve is the audience: a capability names
the executor allowed to spend it, and a router that answered with its own name
would make every executor behind it a valid audience. audienceFor therefore
delegates to the routed executor, and the kernel derives the audience from the
intent rather than from the executor alone. Two executors claiming the same
action are refused at construction — otherwise the one that loses is the one an
operator believed they had deployed.