> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trainy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Enroll a worker cluster

> Connect a Kubernetes worker cluster to a Konduktor organization without interrupting workloads already running on it.

Enrollment connects a Kubernetes cluster to the Konduktor Hub so the Hub can
dispatch work, keep Kueue and JobSet at the organization's supported versions,
and collect workload telemetry.

The process has two parts:

1. An organization administrator creates a one-shot enrollment blob on the
   Hub.
2. A worker-cluster administrator passes that blob to the worker bootstrap.

<Warning>
  Enrollment crosses a trust boundary. Pass an explicit Hub or worker context
  to every command; do not rely on whichever context is currently active.
  Konduktor accepts `--context` and `--kubeconfig` before the subcommand.
</Warning>

## Prerequisites

You need:

* A Konduktor organization and tenant namespace created by Trainy
* An Authentik account with access to the Hub and organization-admin access to
  that tenant namespace
* The Google Cloud CLI and `gke-gcloud-auth-plugin`
* `kubectl`
* Helm v3.17.1 or newer
* Cluster-admin access to the worker cluster
* The `konduktor` CLI
* Outbound HTTPS and WebSocket access from the worker to the Hub endpoints

Set these variables before starting:

```bash theme={null}
WORKER_CONTEXT=forecasting-us
TENANT_NS=forecasting
WORKER=forecasting-us
WORKLOAD_NAMESPACES=("$TENANT_NS")
```

The worker name must be unique inside the tenant namespace. Another
organization may use the same name. `WORKLOAD_NAMESPACES` must match the
namespace list configured on the Konduktor organization. Do not add unrelated
worker-local namespaces: Konduktor does not manage their LocalQueues.

## Get scoped access to the Hub

Trainy does not distribute a static Hub kubeconfig. Human access uses
Authentik and Google Workforce Identity Federation:

1. Authentik verifies your identity and group memberships.
2. Google exchanges that identity for a short-lived Workforce credential.
3. `gcloud` writes a kubeconfig that uses the federated credential.
4. Kubernetes RBAC grants access in the tenant namespaces associated with that
   identity's organization-admin group memberships.

Your Authentik account must belong to both of these groups:

* `konduktor-hub-users`, which permits discovery of and connection to the Hub
* Your organization's administrator group, which grants Kubernetes access in
  the organization's tenant namespace

Ask your organization owner or Trainy to add the memberships if either is
missing. Isolate the Workforce credentials from any ambient Google credentials
before signing in:

```bash theme={null}
umask 077
HUB_AUTH_DIR="$(mktemp -d "${TMPDIR:-/tmp}/konduktor-hub-auth.XXXXXX")"
chmod 700 "$HUB_AUTH_DIR"
export CLOUDSDK_CONFIG="$HUB_AUTH_DIR/gcloud"
mkdir -m 700 "$CLOUDSDK_CONFIG"
unset CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE
unset GOOGLE_APPLICATION_CREDENTIALS
unset GOOGLE_GHA_CREDS_PATH

WORKFORCE_LOGIN_CONFIG="$HUB_AUTH_DIR/workforce-login.json"

gcloud iam workforce-pools create-login-config \
  locations/global/workforcePools/konduktor-hub/providers/authentik \
  --output-file="$WORKFORCE_LOGIN_CONFIG"

gcloud auth login --login-config="$WORKFORCE_LOGIN_CONFIG"
gcloud config set billing/quota_project trainy-test
```

The browser flow starts on Google Cloud and redirects you to Trainy's Authentik
login. After it succeeds, add the Hub to your normal kubeconfig as a new
context. Save and restore your current context so this step does not silently
retarget later commands:

```bash theme={null}
PREVIOUS_CONTEXT="$(kubectl config current-context 2>/dev/null || true)"
gcloud container clusters get-credentials konduktor-hub \
  --project=trainy-test \
  --region=us-central1 \
  --dns-endpoint
HUB_CONTEXT="$(kubectl config current-context)"

if [[ -n "$PREVIOUS_CONTEXT" ]]; then
  kubectl config use-context "$PREVIOUS_CONTEXT"
fi

printf 'CLOUDSDK_CONFIG=%s\nHUB_CONTEXT=%s\n' \
  "$CLOUDSDK_CONFIG" "$HUB_CONTEXT"
```

Keep this shell open for the rest of the procedure. The exported
`CLOUDSDK_CONFIG` makes the Hub context's exec plugin use this isolated
Workforce identity. If you run a Hub command from another terminal, export the
same `CLOUDSDK_CONFIG` there and unset the three credential override variables
before running it.

If your contexts live in a non-default kubeconfig, also pass
`--kubeconfig /absolute/path/to/config` immediately after `konduktor`, and use
the same path with `kubectl --kubeconfig`. Command-local `--kubeconfig` remains
supported for compatibility, but the root form keeps the target selection in
one consistent place. For example:

```bash theme={null}
konduktor --kubeconfig /absolute/path/to/config \
  --context "$HUB_CONTEXT" cluster list -n "$TENANT_NS"
```

Confirm that the federated identity can connect, has the required enrollment
permission in the intended tenant, and cannot create cluster-scoped
Organizations:

```bash theme={null}
kubectl --context "$HUB_CONTEXT" cluster-info
kubectl --context "$HUB_CONTEXT" \
  -n "$TENANT_NS" \
  auth can-i create bootstrapenrollments.trainy.ai
kubectl --context "$HUB_CONTEXT" \
  auth can-i create organizations.trainy.ai
```

The namespace-scoped check must print `yes`; the cluster-scoped Organization
check must print `no`. These checks do not prove that the account lacks access
to every other tenant namespace. If namespace isolation must be audited, ask
your organization owner or Trainy to confirm that the account has no unintended
organization-admin group memberships.

## Enroll a new cluster

<Steps>
  <Step title="Confirm both Kubernetes targets">
    Check the local tools, cluster endpoints, and required permissions before
    making changes:

    ```bash theme={null}
    gcloud --version
    gke-gcloud-auth-plugin --version
    kubectl version --client
    helm version --short

    kubectl --context "$HUB_CONTEXT" cluster-info
    kubectl --context "$HUB_CONTEXT" \
      -n "$TENANT_NS" \
      auth can-i create bootstrapenrollments.trainy.ai

    kubectl --context "$WORKER_CONTEXT" cluster-info
    kubectl --context "$WORKER_CONTEXT" auth can-i create namespaces
    ```

    Both permission checks must print `yes`. Organization administrators do not
    need permission to create `Organization` objects. Confirm that Helm reports
    v3.17.1 or newer before creating the one-shot enrollment.
  </Step>

  <Step title="Create the one-shot bootstrap blob">
    Create the bootstrap file atomically inside a private temporary directory,
    and arrange for it to be removed if this shell exits early:

    ```bash theme={null}
    umask 077
    BOOTSTRAP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/konduktor-bootstrap.XXXXXX")"
    chmod 700 "$BOOTSTRAP_DIR"
    BOOTSTRAP_BLOB="$(mktemp "$BOOTSTRAP_DIR/bootstrap.XXXXXX.txt")"

    cleanup_bootstrap() {
      rm -f -- "$BOOTSTRAP_BLOB"
      rmdir -- "$BOOTSTRAP_DIR" 2>/dev/null || true
    }
    trap cleanup_bootstrap EXIT

    konduktor --context "$HUB_CONTEXT" cluster enroll "$WORKER" \
      --ns "$TENANT_NS" \
      > "$BOOTSTRAP_BLOB"
    chmod 600 "$BOOTSTRAP_BLOB"
    ```

    The CLI discovers the Hub, tunnel, and observability endpoints
    automatically. Do not add `--org`, endpoint overrides, or a CA bundle for
    the hosted Hub.

    <Warning>
      The generated blob contains a private signing key. Treat it as a secret,
      transfer it through an approved secure channel, and remove it after the
      enrollment succeeds. Bootstrap carries that key in its process arguments
      while it runs. Generate and run it only from a trusted, single-user
      administration host, not a shared bastion.
    </Warning>
  </Step>

  <Step title="Bootstrap the worker">
    Reconfirm the worker context, then bootstrap using the blob:

    ```bash theme={null}
    kubectl --context "$WORKER_CONTEXT" cluster-info
    konduktor --context "$WORKER_CONTEXT" \
      worker bootstrap "$(cat "$BOOTSTRAP_BLOB")"
    ```

    Review the plan and approve it. Bootstrap installs the worker agent,
    tunnel credentials, log collector, metrics agent, and their supporting
    namespaces and RBAC. It preserves an existing shared VictoriaMetrics
    operator.
  </Step>

  <Step title="Wait for readiness">
    Check enrollment from the Hub:

    ```bash theme={null}
    konduktor --context "$HUB_CONTEXT" \
      cluster status "$WORKER" -n "$TENANT_NS"
    ```

    Do not submit work until all four conditions are `True`:

    | Condition          | Meaning                                                      |
    | ------------------ | ------------------------------------------------------------ |
    | `Registered`       | The one-shot enrollment was redeemed.                        |
    | `Connected`        | The worker tunnel is connected and its heartbeat is current. |
    | `WorkerConfigured` | The worker's Kueue scheduling configuration matches the Hub. |
    | `DispatchReady`    | The Hub can dispatch work to this worker.                    |

    Check the installed components on the worker if a condition does not
    converge:

    ```bash theme={null}
    kubectl --context "$WORKER_CONTEXT" -n trainy-system get pods
    kubectl --context "$WORKER_CONTEXT" -n trainy-byoc-system get pods
    ```
  </Step>

  <Step title="Remove the bootstrap secret">
    Delete the local bootstrap file and any transferred copies using your
    organization's secret-handling policy. On a filesystem where secure
    deletion is supported, remove this copy before leaving the shell:

    ```bash theme={null}
    if command -v shred >/dev/null 2>&1; then
      shred -u -- "$BOOTSTRAP_BLOB" || rm -f -- "$BOOTSTRAP_BLOB"
    else
      rm -f -- "$BOOTSTRAP_BLOB"
    fi

    if [[ -e "$BOOTSTRAP_BLOB" ]]; then
      printf 'Bootstrap secret still exists at %s; cleanup is still armed.\n' \
        "$BOOTSTRAP_BLOB" >&2
    else
      rmdir -- "$BOOTSTRAP_DIR"
      trap - EXIT
    fi
    ```

    The exit trap still unlinks the local file after an error or interruption.
    `shred` cannot guarantee physical overwriting on every filesystem, including
    copy-on-write, journaled, snapshotted, and remote filesystems. The fallback
    `rm` only unlinks the file. Use your organization's approved secret-deletion
    procedure when unlinking is not sufficient.
  </Step>
</Steps>

## Enroll a cluster without stopping existing workloads

Enrollment does not require draining the worker. The worker agent upgrades
Kueue and JobSet in place to the versions selected by the organization, and a
cordon prevents new Hub dispatch while that configuration converges. The
cordon does not suspend or delete work already running on the cluster.

Use this procedure for a worker that already runs Kueue workloads:

<Steps>
  <Step title="Record the running workload invariants">
    Capture the names, UIDs, and status of active JobSets, Jobs, and Kueue
    Workloads before enrollment:

    ```bash theme={null}
    kubectl --context "$WORKER_CONTEXT" -n default get jobsets -o wide
    kubectl --context "$WORKER_CONTEXT" -n default get jobs -o wide
    kubectl --context "$WORKER_CONTEXT" -n default get workloads -o wide

    kubectl --context "$WORKER_CONTEXT" \
      get deploy -A \
      -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.readyReplicas,IMAGE:.spec.template.spec.containers[0].image' \
      | grep -E 'kueue|jobset'
    ```

    Keep this output until the post-enrollment check is complete.
  </Step>

  <Step title="Pause new Hub submissions">
    Pause every path that submits new work through the Hub for this
    organization, including user shells, CI jobs, and schedulers. Wait for any
    work already queued for Hub dispatch to be admitted or withdrawn before
    starting bootstrap.

    This is a short submission freeze, not a worker outage. Workloads already
    running on the worker continue running.

    <Warning>
      Enrollment cannot currently create an initially cordoned WorkerCluster
      atomically. Registration creates it uncordoned, so keep submissions
      paused until the cluster is cordoned and enrollment validation finishes.
    </Warning>
  </Step>

  <Step title="Bootstrap, then immediately cordon Hub dispatch">
    Start the normal bootstrap command. As soon as the `WorkerCluster` appears,
    run this from another trusted terminal that uses the same isolated
    `CLOUDSDK_CONFIG` and Hub context:

    ```bash theme={null}
    export CLOUDSDK_CONFIG=/private/path/printed-in-the-first-terminal/gcloud
    HUB_CONTEXT=the-context-printed-in-the-first-terminal
    TENANT_NS=forecasting
    WORKER=forecasting-us
    unset CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE
    unset GOOGLE_APPLICATION_CREDENTIALS
    unset GOOGLE_GHA_CREDS_PATH

    konduktor --context "$HUB_CONTEXT" \
      cluster cordon "$WORKER" -n "$TENANT_NS"
    ```

    `DispatchReady` remains `False` while the worker is cordoned. Existing
    worker-local workloads continue running. Keep organization submissions
    paused; the post-registration cordon alone cannot guarantee that there was
    no new-dispatch window.
  </Step>

  <Step title="Wait for dependency convergence">
    Watch the Kueue and JobSet Deployments while the worker agent applies the
    organization versions:

    ```bash theme={null}
    kubectl --context "$WORKER_CONTEXT" \
      -n kueue-system rollout status deploy/kueue-controller-manager

    kubectl --context "$WORKER_CONTEXT" \
      -n jobset-system get deploy -w
    ```

    Older Kueue installations may pass through an intermediate compatibility
    release while their CRDs are rewritten to the current storage version. A
    cluster with many historical Workloads can spend several minutes in this
    step.

    Continue checking the recorded active workloads. Their UIDs and running
    status must remain unchanged.
  </Step>

  <Step title="Resolve existing Kueue object ownership">
    A brownfield cluster may already contain objects that the selected
    scheduling preset also manages. In that case, `WorkerConfigured` reports:

    ```text theme={null}
    refuse to adopt existing object not owned by this WorkerCluster
    ```

    Konduktor refuses automatic adoption because it could otherwise overwrite
    customer-owned scheduling policy. See [Adopt existing Kueue and DWS
    objects](#adopt-existing-kueue-and-dws-objects) before continuing.
  </Step>

  <Step title="Verify the old work, then enable dispatch">
    Confirm the recorded JobSet, Job, and Workload UIDs still match and that the
    running Jobs are Ready. Then uncordon the worker:

    ```bash theme={null}
    konduktor --context "$HUB_CONTEXT" \
      cluster cordon "$WORKER" -n "$TENANT_NS" --uncordon

    konduktor --context "$HUB_CONTEXT" \
      cluster status "$WORKER" -n "$TENANT_NS"
    ```

    Finish only when `WorkerConfigured=True` at the current WorkerCluster
    generation and `DispatchReady=True`.
  </Step>

  <Step title="Run a canary through the Hub">
    Submit a short JobSet through the organization's normal `user-queue`, then
    verify that it completes and its logs are available through Konduktor.

    If the organization has several dispatch-ready workers, a normal canary
    proves the pool, not one particular member. To prove exact routing,
    coordinate a short cordon of the other pool members, submit the canary, and
    uncordon them immediately afterward. Cordoning affects only new dispatch.
  </Step>

  <Step title="Resume organization submissions">
    Resume the user, CI, and scheduler submission paths paused before
    bootstrap. Workloads that were already running have remained active
    throughout enrollment.
  </Step>
</Steps>

## Adopt existing Kueue and DWS objects

<Warning>
  Do not add Konduktor ownership labels until you have compared each existing
  object with the scheduling preset. Adoption authorizes the Hub to update and
  later prune that object. If the specifications differ, stop and contact
  Trainy to express the existing policy as WorkerCluster scheduling overlays.
</Warning>

The `gke-dws` preset manages this complete graph:

* `ResourceFlavor/default-flavor`
* `ClusterQueue/cluster-queue`
* `AdmissionCheck/dws-prov`
* `ProvisioningRequestConfig/dws-config`
* `LocalQueue/user-queue` in every organization workload namespace

Inspect the cluster-scoped objects first:

```bash theme={null}
for object in \
  resourceflavors.kueue.x-k8s.io/default-flavor \
  clusterqueues.kueue.x-k8s.io/cluster-queue \
  admissionchecks.kueue.x-k8s.io/dws-prov \
  provisioningrequestconfigs.kueue.x-k8s.io/dws-config
do
  kubectl --context "$WORKER_CONTEXT" get "$object" -o yaml
done
```

Then inspect the namespaced LocalQueue in every namespace listed in
`WORKLOAD_NAMESPACES`:

```bash theme={null}
for namespace in "${WORKLOAD_NAMESPACES[@]}"
do
  kubectl --context "$WORKER_CONTEXT" \
    -n "$namespace" \
    get localqueues.kueue.x-k8s.io/user-queue -o yaml
done
```

Compare the specifications and UIDs of all five object types with the intended
`gke-dws` preset. After confirming they match, assign the cluster-scoped
objects to this WorkerCluster without recreating them:

```bash theme={null}
WORKER_IDENTITY="${#TENANT_NS}-${TENANT_NS}-${WORKER}"

for object in \
  resourceflavors.kueue.x-k8s.io/default-flavor \
  clusterqueues.kueue.x-k8s.io/cluster-queue \
  admissionchecks.kueue.x-k8s.io/dws-prov \
  provisioningrequestconfigs.kueue.x-k8s.io/dws-config
do
  kubectl --context "$WORKER_CONTEXT" label "$object" \
    trainy.ai/managed-by=worker-scheduling \
    trainy.ai/worker-cluster="$WORKER_IDENTITY" \
    --overwrite
done
```

Apply the same ownership labels to each namespaced LocalQueue:

```bash theme={null}
for namespace in "${WORKLOAD_NAMESPACES[@]}"
do
  kubectl --context "$WORKER_CONTEXT" \
    -n "$namespace" \
    label localqueues.kueue.x-k8s.io/user-queue \
    trainy.ai/managed-by=worker-scheduling \
    trainy.ai/worker-cluster="$WORKER_IDENTITY" \
    --overwrite
done
```

Select the DWS preset on the Hub:

```bash theme={null}
kubectl --context "$HUB_CONTEXT" \
  -n "$TENANT_NS" \
  patch workercluster "$WORKER" \
  --type merge \
  -p '{"spec":{"scheduling":{"preset":"gke-dws"}}}'
```

Wait for `WorkerConfigured=True` before uncordoning. Recheck every cluster- and
namespace-scoped UID to confirm that adoption updated the existing objects
rather than replacing them.

## Troubleshooting

### `Registered=False`

Check whether the one-shot enrollment was redeemed:

```bash theme={null}
kubectl --context "$HUB_CONTEXT" \
  -n "$TENANT_NS" \
  get bootstrapenrollment "$WORKER" -o yaml
```

An empty `status.usedAt` means the worker did not complete registration. Check
the bootstrap output and worker access to the Hub API.

### `Connected=False`

Check the worker agent and outbound access to the Hub tunnel:

```bash theme={null}
kubectl --context "$WORKER_CONTEXT" \
  -n trainy-system logs deploy/trainy-worker-agent
```

### `WorkerConfigured=False`

Read the condition message from `konduktor cluster status`. An adoption refusal
requires the review described above. Other failures usually identify the Kueue
object or API that did not apply.

### `DispatchReady=False`

Confirm the worker is connected, configured at the current generation, and not
cordoned. `DispatchReady` becomes `True` only after the Hub's MultiKueue member
is active.

### Observability says `NoTrafficYet`

`ObservabilityIngestHealthy=Unknown` does not make the four enrollment
conditions false. Run a canary long enough for logs and metrics to be scraped,
then verify both through the Hub. A worker using the gateway transport can have
queryable telemetry while the tunnel-traffic health condition still reports
`NoTrafficYet`.

## Decommission or move a worker

Stop new Hub dispatch before maintenance:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" \
  cluster cordon "$WORKER" -n "$TENANT_NS"
```

To remove the Hub enrollment, then remove Trainy-installed worker components:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" \
  cluster decommission "$WORKER" -n "$TENANT_NS" --yes

konduktor --context "$WORKER_CONTEXT" worker reset --dry-run
konduktor --context "$WORKER_CONTEXT" worker reset
```

`worker reset` leaves customer workloads, Kueue and JobSet objects, cert-manager,
and a possibly shared VictoriaMetrics operator in place. Enrolling the cluster
again requires a new one-shot bootstrap command.