Enroll a worker cluster

Enrollment connects a Kubernetes cluster to the Konduktor Hub so the Hub can dispatch work, keep Kueue and JobSet at the organization’s supported versions, and collect workload telemetry. The process has two parts:

An organization administrator creates a one-shot enrollment blob on the Hub.
A worker-cluster administrator passes that blob to the worker bootstrap.

Enrollment crosses a trust boundary. Pass an explicit Hub or worker context to every command; do not rely on whichever context is currently active. Konduktor accepts --context and --kubeconfig before the subcommand.

Prerequisites

You need:

A Konduktor organization and tenant namespace created by Trainy
An accepted WorkOS invitation to your Konduktor organization and organization-admin access to that tenant namespace
The Google Cloud CLI and gke-gcloud-auth-plugin
kubectl
Helm v3.17.1 or newer
Cluster-admin access to the worker cluster
The konduktor CLI
Outbound HTTPS and WebSocket access from the worker to the Hub endpoints

Set these variables before starting. Replace the example values with your worker context, assigned tenant namespace, and chosen worker name:

WORKER_CONTEXT=example-worker-us
TENANT_NS=example-team
WORKER=example-worker-us
WORKLOAD_NAMESPACES=("$TENANT_NS")

The worker name must be unique inside the tenant namespace. Another organization may use the same name. WORKLOAD_NAMESPACES must match the namespace list configured on the Konduktor organization. Do not add unrelated worker-local namespaces: Konduktor does not manage their LocalQueues.

Get scoped access to the Hub

konduktor login --namespace "$TENANT_NS"
HUB_CONTEXT='<Hub Kubernetes context printed by login>'

Complete the WorkOS device approval in your browser. The CLI discovers your assigned namespaces, selects the WorkOS organization for TENANT_NS, and exchanges its access token for a short-lived Google Workforce credential. Google does not start another browser login. The CLI creates a Hub Kubernetes context and makes it current after successful setup. Your organization owner or Trainy must invite you and assign the tenant namespace before login can configure it. Enrollment also requires permission to create BootstrapEnrollment objects in that namespace. Signing in does not grant that permission by itself. Use the printed Hub context explicitly for the rest of this procedure. Each context records its own organization-specific credential source under ~/.konduktor/auth/; another terminal can use the same kubeconfig without exporting CLOUDSDK_CONFIG. Keep the CLI installed in the Python environment used for login so the credential source can refresh your session. If that environment changes, run konduktor login again. If your contexts live in a non-default kubeconfig, pass its path before the login command and use the same path for later commands:

konduktor --kubeconfig /absolute/path/to/config login --namespace "$TENANT_NS"
konduktor --kubeconfig /absolute/path/to/config \
  --context "$HUB_CONTEXT" cluster list -n "$TENANT_NS"

Use kubectl --kubeconfig /absolute/path/to/config for the checks below if you selected a non-default file. Confirm that the federated identity can connect, has the required enrollment permission in the intended tenant, and cannot create cluster-scoped Organizations:

kubectl --context "$HUB_CONTEXT" cluster-info
kubectl --context "$HUB_CONTEXT" \
  -n "$TENANT_NS" \
  auth can-i create bootstrapenrollments.trainy.ai
kubectl --context "$HUB_CONTEXT" \
  auth can-i create organizations.trainy.ai

The namespace-scoped check must print yes; the cluster-scoped Organization check must print no. These checks do not prove that your account lacks access to every other tenant. Ask your organization owner or Trainy to confirm your organization memberships if you need an access audit.

Enroll a new cluster

Confirm both Kubernetes targets

Check the local tools, cluster endpoints, and required permissions before making changes:

gcloud --version
gke-gcloud-auth-plugin --version
kubectl version --client
helm version --short

kubectl --context "$HUB_CONTEXT" cluster-info
kubectl --context "$HUB_CONTEXT" \
  -n "$TENANT_NS" \
  auth can-i create bootstrapenrollments.trainy.ai

kubectl --context "$WORKER_CONTEXT" cluster-info
kubectl --context "$WORKER_CONTEXT" auth can-i create namespaces

Both permission checks must print yes. Organization administrators do not need permission to create Organization objects. Confirm that Helm reports v3.17.1 or newer before creating the one-shot enrollment.

Create the one-shot bootstrap blob

Create the bootstrap file atomically inside a private temporary directory, and arrange for it to be removed if this shell exits early:

umask 077
BOOTSTRAP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/konduktor-bootstrap.XXXXXX")"
chmod 700 "$BOOTSTRAP_DIR"
BOOTSTRAP_BLOB="$(mktemp "$BOOTSTRAP_DIR/bootstrap.XXXXXX.txt")"

cleanup_bootstrap() {
  rm -f -- "$BOOTSTRAP_BLOB"
  rmdir -- "$BOOTSTRAP_DIR" 2>/dev/null || true
}
trap cleanup_bootstrap EXIT

konduktor --context "$HUB_CONTEXT" cluster enroll "$WORKER" \
  --ns "$TENANT_NS" \
  > "$BOOTSTRAP_BLOB"
chmod 600 "$BOOTSTRAP_BLOB"

The CLI discovers the Hub, tunnel, and observability endpoints automatically. Do not add --org, endpoint overrides, or a CA bundle for the hosted Hub.

The generated blob contains a private signing key. Treat it as a secret, transfer it through an approved secure channel, and remove it after the enrollment succeeds. Bootstrap carries that key in its process arguments while it runs. Generate and run it only from a trusted, single-user administration host, not a shared bastion.

Bootstrap the worker

Reconfirm the worker context, then bootstrap using the blob:

kubectl --context "$WORKER_CONTEXT" cluster-info
konduktor --context "$WORKER_CONTEXT" \
  worker bootstrap "$(cat "$BOOTSTRAP_BLOB")"

Review the plan and approve it. Bootstrap installs the worker agent, tunnel credentials, log collector, metrics agent, and their supporting namespaces and RBAC. It preserves an existing shared VictoriaMetrics operator.

Wait for readiness

Check enrollment from the Hub:

konduktor --context "$HUB_CONTEXT" \
  cluster status "$WORKER" -n "$TENANT_NS"

Do not submit work until all four conditions are True:

Condition	Meaning
`Registered`	The one-shot enrollment was redeemed.
`Connected`	The worker tunnel is connected and its heartbeat is current.
`WorkerConfigured`	The worker’s Kueue scheduling configuration matches the Hub.
`DispatchReady`	The Hub can dispatch work to this worker.

Check the installed components on the worker if a condition does not converge:

kubectl --context "$WORKER_CONTEXT" -n trainy-system get pods
kubectl --context "$WORKER_CONTEXT" -n trainy-byoc-system get pods

Remove the bootstrap secret

Delete the local bootstrap file and any transferred copies using your organization’s secret-handling policy. On a filesystem where secure deletion is supported, remove this copy before leaving the shell:

if command -v shred >/dev/null 2>&1; then
  shred -u -- "$BOOTSTRAP_BLOB" || rm -f -- "$BOOTSTRAP_BLOB"
else
  rm -f -- "$BOOTSTRAP_BLOB"
fi

if [[ -e "$BOOTSTRAP_BLOB" ]]; then
  printf 'Bootstrap secret still exists at %s; cleanup is still armed.\n' \
    "$BOOTSTRAP_BLOB" >&2
else
  rmdir -- "$BOOTSTRAP_DIR"
  trap - EXIT
fi

The exit trap still unlinks the local file after an error or interruption. shred cannot guarantee physical overwriting on every filesystem, including copy-on-write, journaled, snapshotted, and remote filesystems. The fallback rm only unlinks the file. Use your organization’s approved secret-deletion procedure when unlinking is not sufficient.

Enroll a cluster without stopping existing workloads

Enrollment does not require draining the worker. The worker agent upgrades Kueue and JobSet in place to the versions selected by the organization, and a cordon prevents new Hub dispatch while that configuration converges. The cordon does not suspend or delete work already running on the cluster. Use this procedure for a worker that already runs Kueue workloads:

Record the running workload invariants

Capture the names, UIDs, and status of active JobSets, Jobs, and Kueue Workloads before enrollment:

kubectl --context "$WORKER_CONTEXT" -n default get jobsets -o wide
kubectl --context "$WORKER_CONTEXT" -n default get jobs -o wide
kubectl --context "$WORKER_CONTEXT" -n default get workloads -o wide

kubectl --context "$WORKER_CONTEXT" \
  get deploy -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.readyReplicas,IMAGE:.spec.template.spec.containers[0].image' \
  | grep -E 'kueue|jobset'

Keep this output until the post-enrollment check is complete.

Pause new Hub submissions

Pause every path that submits new work through the Hub for this organization, including user shells, CI jobs, and schedulers. Wait for any work already queued for Hub dispatch to be admitted or withdrawn before starting bootstrap.This is a short submission freeze, not a worker outage. Workloads already running on the worker continue running.

Enrollment cannot currently create an initially cordoned WorkerCluster atomically. Registration creates it uncordoned, so keep submissions paused until the cluster is cordoned and enrollment validation finishes.

Bootstrap, then immediately cordon Hub dispatch

Start the normal bootstrap command. As soon as the WorkerCluster appears, run this from another trusted terminal that uses the same kubeconfig and printed Hub context:

HUB_CONTEXT=the-context-printed-in-the-first-terminal
TENANT_NS=example-team
WORKER=example-worker-us

konduktor --context "$HUB_CONTEXT" \
  cluster cordon "$WORKER" -n "$TENANT_NS"

DispatchReady remains False while the worker is cordoned. Existing worker-local workloads continue running. Keep organization submissions paused; the post-registration cordon alone cannot guarantee that there was no new-dispatch window.

Wait for dependency convergence

Watch the Kueue and JobSet Deployments while the worker agent applies the organization versions:

kubectl --context "$WORKER_CONTEXT" \
  -n kueue-system rollout status deploy/kueue-controller-manager

kubectl --context "$WORKER_CONTEXT" \
  -n jobset-system get deploy -w

Older Kueue installations may pass through an intermediate compatibility release while their CRDs are rewritten to the current storage version. A cluster with many historical Workloads can spend several minutes in this step.Continue checking the recorded active workloads. Their UIDs and running status must remain unchanged.

Resolve existing Kueue object ownership

A brownfield cluster may already contain objects that the selected scheduling preset also manages. In that case, WorkerConfigured reports:

refuse to adopt existing object not owned by this WorkerCluster

Konduktor refuses automatic adoption because it could otherwise overwrite customer-owned scheduling policy. See Adopt existing Kueue and DWS objects before continuing.

Verify the old work, then enable dispatch

Confirm the recorded JobSet, Job, and Workload UIDs still match and that the running Jobs are Ready. Then uncordon the worker:

konduktor --context "$HUB_CONTEXT" \
  cluster cordon "$WORKER" -n "$TENANT_NS" --uncordon

konduktor --context "$HUB_CONTEXT" \
  cluster status "$WORKER" -n "$TENANT_NS"

Finish only when WorkerConfigured=True at the current WorkerCluster generation and DispatchReady=True.

Run a canary through the Hub

Submit a short JobSet through the organization’s normal user-queue, then verify that it completes and its logs are available through Konduktor.If the organization has several dispatch-ready workers, a normal canary proves the pool, not one particular member. To prove exact routing, coordinate a short cordon of the other pool members, submit the canary, and uncordon them immediately afterward. Cordoning affects only new dispatch.

Resume organization submissions

Resume the user, CI, and scheduler submission paths paused before bootstrap. Workloads that were already running have remained active throughout enrollment.

Adopt existing Kueue and DWS objects

Do not add Konduktor ownership labels until you have compared each existing object with the scheduling preset. Adoption authorizes the Hub to update and later prune that object. If the specifications differ, stop and contact Trainy to express the existing policy as WorkerCluster scheduling overlays.

The gke-dws preset manages this complete graph:

ResourceFlavor/default-flavor
ClusterQueue/cluster-queue
AdmissionCheck/dws-prov
ProvisioningRequestConfig/dws-config
LocalQueue/user-queue in every organization workload namespace

Inspect the cluster-scoped objects first:

for object in \
  resourceflavors.kueue.x-k8s.io/default-flavor \
  clusterqueues.kueue.x-k8s.io/cluster-queue \
  admissionchecks.kueue.x-k8s.io/dws-prov \
  provisioningrequestconfigs.kueue.x-k8s.io/dws-config
do
  kubectl --context "$WORKER_CONTEXT" get "$object" -o yaml
done

Then inspect the namespaced LocalQueue in every namespace listed in WORKLOAD_NAMESPACES:

for namespace in "${WORKLOAD_NAMESPACES[@]}"
do
  kubectl --context "$WORKER_CONTEXT" \
    -n "$namespace" \
    get localqueues.kueue.x-k8s.io/user-queue -o yaml
done

Compare the specifications and UIDs of all five object types with the intended gke-dws preset. After confirming they match, assign the cluster-scoped objects to this WorkerCluster without recreating them:

WORKER_IDENTITY="${#TENANT_NS}-${TENANT_NS}-${WORKER}"

for object in \
  resourceflavors.kueue.x-k8s.io/default-flavor \
  clusterqueues.kueue.x-k8s.io/cluster-queue \
  admissionchecks.kueue.x-k8s.io/dws-prov \
  provisioningrequestconfigs.kueue.x-k8s.io/dws-config
do
  kubectl --context "$WORKER_CONTEXT" label "$object" \
    trainy.ai/managed-by=worker-scheduling \
    trainy.ai/worker-cluster="$WORKER_IDENTITY" \
    --overwrite
done

Apply the same ownership labels to each namespaced LocalQueue:

for namespace in "${WORKLOAD_NAMESPACES[@]}"
do
  kubectl --context "$WORKER_CONTEXT" \
    -n "$namespace" \
    label localqueues.kueue.x-k8s.io/user-queue \
    trainy.ai/managed-by=worker-scheduling \
    trainy.ai/worker-cluster="$WORKER_IDENTITY" \
    --overwrite
done

Select the DWS preset on the Hub:

kubectl --context "$HUB_CONTEXT" \
  -n "$TENANT_NS" \
  patch workercluster "$WORKER" \
  --type merge \
  -p '{"spec":{"scheduling":{"preset":"gke-dws"}}}'

Wait for WorkerConfigured=True before uncordoning. Recheck every cluster- and namespace-scoped UID to confirm that adoption updated the existing objects rather than replacing them.

Troubleshooting

`Registered=False`

Check whether the one-shot enrollment was redeemed:

kubectl --context "$HUB_CONTEXT" \
  -n "$TENANT_NS" \
  get bootstrapenrollment "$WORKER" -o yaml

An empty status.usedAt means the worker did not complete registration. Check the bootstrap output and worker access to the Hub API.

`Connected=False`

Check the worker agent and outbound access to the Hub tunnel:

kubectl --context "$WORKER_CONTEXT" \
  -n trainy-system logs deploy/trainy-worker-agent

`WorkerConfigured=False`

Read the condition message from konduktor cluster status. An adoption refusal requires the review described above. Other failures usually identify the Kueue object or API that did not apply.

`DispatchReady=False`

Confirm the worker is connected, configured at the current generation, and not cordoned. DispatchReady becomes True only after the Hub’s MultiKueue member is active.

Observability says `NoTrafficYet`

ObservabilityIngestHealthy=Unknown does not make the four enrollment conditions false. Run a canary long enough for logs and metrics to be scraped, then verify both through the Hub. A worker using the gateway transport can have queryable telemetry while the tunnel-traffic health condition still reports NoTrafficYet.

Decommission or move a worker

Stop new Hub dispatch before maintenance:

konduktor --context "$HUB_CONTEXT" \
  cluster cordon "$WORKER" -n "$TENANT_NS"

To remove the Hub enrollment, then remove Trainy-installed worker components:

konduktor --context "$HUB_CONTEXT" \
  cluster decommission "$WORKER" -n "$TENANT_NS" --yes

konduktor --context "$WORKER_CONTEXT" worker reset --dry-run
konduktor --context "$WORKER_CONTEXT" worker reset

worker reset leaves customer workloads, Kueue and JobSet objects, cert-manager, and a possibly shared VictoriaMetrics operator in place. Enrolling the cluster again requires a new one-shot bootstrap command.

Get Started

CLI

User Guides

Cluster Operators

Prerequisites

Get scoped access to the Hub

Enroll a new cluster

Enroll a cluster without stopping existing workloads

Adopt existing Kueue and DWS objects

Troubleshooting

`Registered=False`

`Connected=False`

`WorkerConfigured=False`

`DispatchReady=False`

Observability says `NoTrafficYet`

Decommission or move a worker

​Prerequisites

​Get scoped access to the Hub

​Enroll a new cluster

​Enroll a cluster without stopping existing workloads

​Adopt existing Kueue and DWS objects

​Troubleshooting

​Registered=False

​Connected=False

​WorkerConfigured=False

​DispatchReady=False

​Observability says NoTrafficYet

​Decommission or move a worker

Prerequisites

Get scoped access to the Hub

Enroll a new cluster

Enroll a cluster without stopping existing workloads

Adopt existing Kueue and DWS objects

Troubleshooting

`Registered=False`

`Connected=False`

`WorkerConfigured=False`

`DispatchReady=False`

Observability says `NoTrafficYet`

Decommission or move a worker