> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trainy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run commands in a Hub job (experimental)

> Use konduktor exec and shell to inspect a running job through the Hub.

Use `konduktor exec` to run a command in a job's container, or `konduktor shell`
to open an interactive terminal. Both commands connect through the Hub using
your selected Hub Kubernetes credential. You do not need a worker kubeconfig.

> [!NOTE]
>
> **Experimental.** These commands may change before they stabilize.
> Hub job execution is disabled by default. Ask your organization administrator
> or Trainy whether execution is enabled on your Hub and worker before using
> these commands. Access to job status or logs alone does not grant execution
> permission.

## Select a running job

Follow [Submit jobs through Trainy Hub](https://docs.trainy.ai/hub-quickstart.md) to install the CLI,
sign in, and select your assigned Hub namespace. You need an invitation and
assigned namespace before login can set up Hub access.

Set the context printed by login, then list your jobs:

```bash theme={null}
export HUB_CONTEXT='your-hub-context-from-login'
konduktor --context "$HUB_CONTEXT" status
```

Use the full job ID from `status`. The job must have exactly one running Pod.
Queued, completed, and suspended jobs have no running target. Jobs with multiple
eligible running Pods are rejected because rank selection is not supported.

```bash theme={null}
export JOB_ID='your-running-job-id'
```

If you need a job to inspect, save this as `shell-demo.yaml`. Use your
organization's assigned queue if it differs from `user-queue`:

```yaml theme={null}
name: shell-demo
num_nodes: 1
resources:
  cpus: 1
  memory: 1
  image_id: ubuntu:22.04
  labels:
    kueue.x-k8s.io/queue-name: user-queue
run: |
  echo "Ready for exec"
  sleep 3600
```

Submit it, save the generated job ID as `JOB_ID`, and wait for the job to run:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" launch shell-demo.yaml --detach-run
konduktor --context "$HUB_CONTEXT" status
```

## Run a command

Put the command and its arguments after `--`:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" exec "$JOB_ID" -- hostname
```

The output is the worker Pod's hostname. For a GPU job whose image includes
`nvidia-smi`, inspect the GPUs with:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" exec "$JOB_ID" -- nvidia-smi
```

`exec` returns the remote command's exit code. It passes arguments directly to
the executable. To use pipes, variable expansion, or other shell syntax in the
container, invoke a shell explicitly and quote the script:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" exec "$JOB_ID" -- sh -c 'echo "$HOSTNAME"; pwd'
```

To forward standard input, add `--stdin` before the job ID:

```bash theme={null}
printf 'hello from stdin\n' | konduktor --context "$HUB_CONTEXT" exec --stdin "$JOB_ID" -- cat
```

The command prints `hello from stdin` and exits when input ends.

## Open a shell

From an interactive terminal, run:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" shell "$JOB_ID"
```

The default shell is `/bin/bash`. If the image has only `/bin/sh`, select it:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" shell "$JOB_ID" --shell /bin/sh
```

The shell must exist in the job image. Type `exit` to leave it. Terminal resize
updates reach the remote shell, and the CLI restores your local terminal settings
when the session ends. `exec --tty` also allocates a terminal and forwards input.
Both `shell` and `exec --tty` require interactive standard input and output.

To use another Hub namespace you can access, pass `--namespace`:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" shell "$JOB_ID" --namespace your-assigned-namespace
```

If you launched the demo job, suspend it when you finish:

```bash theme={null}
konduktor --context "$HUB_CONTEXT" stop "$JOB_ID"
```

## Resolve connection errors

| Message or symptom | Action |
| - | - |
| Select the Hub context created by login | Run `konduktor login`, then use the printed context with `--context` before the command. |
| Job not found | Check the full job ID and selected Hub namespace. |
| Credential expired | Run `konduktor login` again. |
| Lacks execution permission | Ask your organization administrator for execution access in the job's namespace. The Hub requires JobSet read permission and `create` on `pods/exec`. |
| Execution is not enabled, or worker capability is unavailable | Ask your administrator to check Hub enablement, public routing, and worker support. |
| Job needs exactly one running Pod | Check `status`. Wait for a pending job to start. Multiple running Pods are unsupported. |
| Execution capacity is full | Retry after another session ends. |
| Shell executable is missing | Select an executable present in the image with `--shell`. |
| A shell or `--tty` requires an interactive terminal | Run from a terminal, or use `exec` without `--tty` for scripts and redirected output. |

Sessions can end when credentials expire, access is revoked, or job placement
changes. Hub, worker, and edge connection limits also bound session duration.
If a connection ends before an exit status arrives, the CLI reports an error.
Check the job before retrying a command that changes data.

For the complete option list, see the
[`exec` and `shell` CLI reference](https://docs.trainy.ai/cli-reference#konduktor-exec).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.