> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trainy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Submit jobs through Trainy Hub

> Move from direct cluster submission to Hub login, cluster discovery, jobs, and logs.

> [!NOTE]
>
> Trainy Hub is in **private preview**. Ask your organization owner or
> [Trainy](mailto:founders@trainy.ai) for an invitation and an assigned Hub namespace.

Use this guide if you already submit Konduktor jobs directly to a cluster.
With Hub, you submit to one Hub context and namespace. Hub dispatches the job to
an eligible worker cluster in your organization. You do not need that worker's
kubeconfig to submit jobs or read their application logs.

## Sign in and select your workspace

Install or upgrade the CLI in the Python environment you use for jobs:

```bash theme={null}
pip install --upgrade konduktor-nightly
```

Install `kubectl`, the Google Cloud CLI (`gcloud`), and
[`gke-gcloud-auth-plugin`](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin).
Then sign in:

```bash theme={null}
konduktor login
```

Complete the sign-in in your browser. Login discovers the namespaces your account
can access. If several are available, choose one at the prompt. The selected
namespace determines the organization used for Hub credentials.

Successful login prints the Hub context and namespace, makes that context current,
and checks that you can submit JobSets. If no namespace is assigned, ask your
organization owner or Trainy for access.

Save the exact values printed by login. Replace the example values below:

```bash theme={null}
export HUB_CONTEXT='your-hub-context-from-login'
export HUB_NAMESPACE='your-assigned-namespace'
```

Confirm them before submitting:

```bash theme={null}
kubectl config current-context
kubectl --context "$HUB_CONTEXT" config view --minify \
  -o 'jsonpath={.contexts[0].context.namespace}'
konduktor --context "$HUB_CONTEXT" status
```

The `Cluster:` line in `status` identifies your Kubernetes context. Check that
`Namespace:` matches your Hub namespace. It does not identify the job's destination
worker cluster.

This guide uses explicit `--context` values so another tool changing your current
context does not redirect your commands. Put this global option before the command.
If `kubernetes.allowed_contexts` in `~/.konduktor/config.yaml` pins an old worker
context, explicit `--context` overrides it. Login also warns about that setting.

To choose a different assigned namespace later, run login again:

```bash theme={null}
konduktor login --namespace "$HUB_NAMESPACE"
```

To sign in without changing or creating a Kubernetes context, use
`konduktor login --no-hub-context`.

### Sign in from a remote VM

Run this on the VM, with its assigned namespace:

```bash theme={null}
export HUB_NAMESPACE='your-assigned-namespace'
konduktor login --namespace "$HUB_NAMESPACE" --no-browser
```

Open the printed URL on a computer with a browser and complete the device-code
sign-in. Leave the command running on the VM until context setup finishes.
Credentials remain on the VM. Current Hub context setup uses the signed-in session
for Google Workforce credentials; it does not require a browser on the VM.

### Check and renew your session

```bash theme={null}
konduktor whoami
```

`expires` describes the cached bearer token, rather than a timer that requires
hourly browser login. When a command needs credentials near or after expiry, the
CLI uses its refresh token to renew the session without a browser. `whoami` reads
the cache and does not trigger refresh. Token lifetimes depend on the identity
provider. If refresh expires or is revoked, run `konduktor login` again, using
`--no-browser` on a remote VM.

## List clusters and check readiness

List workers registered in your Hub namespace:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" cluster list -n "$HUB_NAMESPACE"
```

Check `REGISTERED`, `CONNECTED`, `CONFIGURED`, and `DISPATCHREADY`. A worker ready
for dispatch reports `True` for these conditions and is not `CORDONED`. Readiness
does not guarantee free capacity or immediate admission for your resource request.

For condition messages and recent events on a worker, replace the name below with
one from the list:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" cluster status worker-name -n "$HUB_NAMESPACE"
```

Repeat the namespace-scoped list for each namespace you can access. The
[Hub console](https://hub.trainy.ai/console) also lets you select an accessible
workspace and inspect its clusters.

Platform administrators with permission to list workers across namespaces can use:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" cluster list -A
```

`-A` and `--all-namespaces` go after `list`. This requests a cluster-wide listing;
it is not a shortcut for listing only your account's memberships. Tenant-scoped
accounts may receive a permission error.

## Adapt your task and submit

Keep your existing task's image, resource requests, environment, and `run` script.
Use the queue assigned by your organization, normally `user-queue`. Review any
worker-specific node selectors, PVC mounts, host paths, and secret references
before you reuse them. A PVC or path on your old cluster is not copied to another
worker by Hub.

For a first CPU job, save this as `hello-hub.yaml`:

```yaml theme={null}
name: hello-hub
num_nodes: 1
resources:
  cpus: 1
  memory: 1
  image_id: ubuntu:22.04
  labels:
    kueue.x-k8s.io/queue-name: user-queue
run: |
  echo "Hello from Hub"
```

Submit through the Hub context:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" launch hello-hub.yaml --detach-run
```

Confirm the launch prompt. `--detach-run` submits without waiting for log streaming.
The CLI prints a generated job ID such as
`hello-hub-ab12`. Save that full ID for later commands:

```bash theme={null}
export JOB_ID='hello-hub-ab12'
```

The Hub context chooses where you submit; it does not pin execution to one worker.
Hub dispatches among eligible workers according to queues and resource availability.
Ask your organization administrator about placement requirements for workloads
that need a specific cluster's storage or hardware.

## View jobs and logs

```bash theme={null}
konduktor --context "$HUB_CONTEXT" status
konduktor --context "$HUB_CONTEXT" logs --no-follow "$JOB_ID"
konduktor --context "$HUB_CONTEXT" logs "$JOB_ID"
```

`status` lists your jobs in the selected Hub namespace. Use `status --all-users`
to include other users' jobs there. `logs` follows by default; `--no-follow` prints
available logs and exits. Use `--node-rank 1` to read rank 1 of a multi-node job.
Hub reads application logs from workers without switching your kubecontext.

Open the [Hub console](https://hub.trainy.ai/console), select your workspace, and
open a job to inspect its selected worker, scheduling conditions, and admission
checks. The console's **Clusters** view shows connection and dispatch readiness.
Your console sign-in is separate from the CLI session.

## Make private images and other dependencies available

Create referenced Secrets and ConfigMaps in the **Hub job's namespace**, even if
they already exist in your old worker cluster's `default` namespace. Hub does not
import those worker-only objects. This applies to image-pull credentials,
container environment references, and Secret or ConfigMap volumes, including
projected volumes and init-container references.

For a private image, obtain a Docker configuration JSON file with registry
authentication entries. A file containing only a local credential-helper setting
does not supply credentials to Kubernetes. Create the Hub secret from that file:

```bash theme={null}
kubectl --context "$HUB_CONTEXT" -n "$HUB_NAMESPACE" \
  create secret generic registry-pull \
  --type=kubernetes.io/dockerconfigjson \
  --from-file=.dockerconfigjson=./registry-auth.json
```

For registry-specific credential creation, see
[private container registry setup](https://docs.trainy.ai/setup.md#optional-setup-private-container-registry-credentials).
Use the Hub context and namespace for those commands.

Merge this into `~/.konduktor/config.yaml` and keep your other settings:

```yaml theme={null}
kubernetes:
  pod_config:
    spec:
      imagePullSecrets:
        - name: registry-pull
```

Set your task's `resources.image_id` to the private image. Before launch, check
that each referenced dependency exists in the Hub namespace:

```bash theme={null}
kubectl --context "$HUB_CONTEXT" -n "$HUB_NAMESPACE" get secret registry-pull
```

During dependency sync, Hub mirrors referenced Secrets and ConfigMaps into the
same namespace on eligible workers before dispatch proceeds. The Hub object is
the source; matching secret names on different workers do not prove that their
credentials match. Treat this as workload dependency sync, not a continuous
registry-secret distribution service. For credential rotation, update the Hub
source and coordinate new submissions with your administrator; do not assume an
edit updates every existing job or worker copy immediately.

## Diagnose a queued job or empty logs

Start with the job's conditions and admission checks in the console. For a CLI
inspection, list the Hub Workloads, find the one whose owner is your JobSet, and
inspect its details:

```bash theme={null}
kubectl --context "$HUB_CONTEXT" -n "$HUB_NAMESPACE" get workloads
kubectl --context "$HUB_CONTEXT" -n "$HUB_NAMESPACE" describe workload workload-name
```

* `missing source dependencies: Secret/...` or `ConfigMap/...` means the source
  object is missing from the Hub namespace. Create it there, then check admission
  again. Creating it only in a worker's `default` namespace does not fix this.
* An offline worker or a failed mirror can hold dependency sync pending. Check
  worker readiness and share the condition message with your administrator.
* A dispatched job with no application logs may still be pulling its image or
  starting its container. Ask an administrator to inspect the worker pod's events
  for `ErrImagePull`, `ImagePullBackOff`, or missing configuration. Invalid registry
  credentials require a valid Hub source secret, not another CLI sign-in.
* If logs report a missing job in `default`, check the context and namespace.
  Use the same explicit Hub context used for launch and the full generated job ID.
* If the console returns 503, refresh and retry. If it persists, share the time,
  workspace, and job ID with Trainy. If the console says your session ended, sign
  in again there.

To continue submitting directly to an existing worker, pass its context explicitly:

```bash theme={null}
konduktor --context your-worker-context launch existing-task.yaml
```

That command uses the worker context's namespace and its existing credentials.
See [direct cluster authentication](https://docs.trainy.ai/authentication.md) and the
[CLI reference](https://docs.trainy.ai/cli-reference.md) for the remaining commands.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.