konduktor status
Shows list of all the jobs with optional filtering and pagination.
Usage
--all-users, -u
Show all jobs, including those not owned by the current user.--limit, -l INTEGER
Maximum number of jobs to display (e.g., —limit 100)--after TEXT
Show jobs created after this timestamp (e.g., —after “08/06/25 03:54PM”, —after “08/06/25”, —after “03:54PM”)--before TEXT
Show jobs created before this timestamp (e.g., —before “08/06/25 03:54PM”, —before “08/06/25”, —before “03:54PM”)
- When using —before or —after timestamps, “08/06/25” is equivalent to “08/06/25 00:00”.
- “03:53PM” is equivalent to “03:53:00PM”.
- Timestamps shown in “konduktor status” are truncated and are in the local timezone. Example: “03:53:55PM” → “03:53PM” would show up in —after “03:53PM” but not in —before “03:53PM”.
konduktor logs
Retrieve/tail the log of a job.
Usage
JOB_ID
--status
[DEPRECATED] If specified, do not show logs but exit with a status code for the job’s status: 0 for succeeded, or 1 for all other statuses.--follow, --no-follow
Follow the logs of a job. If —no-follow is specified, print the log so far and exit. (default: —follow)--num-lines, --num_lines, -n INTEGER
The number of lines to display from the end of the log file. Default is -1 (no limit). (default:-1)--node-rank, --node_rank, -N INTEGER
The node rank to tail logs from. (default:0)--start-offset, --start_offset TEXT
Choose how much time from now to look back in logs. Examples: 30s, 5m, 2h, 1d. Default is 1h. Note: currently only applies when streaming (default —follow). With —no-follow, all available logs are returned. (default:1h)
konduktor launch
Launch a task.
Usage
ENTRYPOINT...
-
--dryrun
If True, do not actually run the job. -
--detach-run, -d
If True, as soon as a job is submitted, return from this call and do not stream execution logs. -
--name, -n TEXT
Task name. Overrides the “name” config in the YAML if both are supplied. -
--workdir DIRECTORY
If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the “workdir” config in the YAML if both are supplied. -
--cloud TEXT
The cloud to use. If specified, overrides the “resources.cloud” config. Passing “none” resets the config. [defunct] currently only supports a single cloud -
--num-nodes INTEGER
Number of nodes to execute the task on. Overrides the “num_nodes” config in the YAML if both are supplied. -
--max-restarts INTEGER
Maximum number of jobset restarts allowed. Overrides YAML.Overrides the “max_restarts” config in the YAML if both are supplied. -
--completions INTEGER
Number of successful completions required. Overrides YAML.Overrides the “completions” config in the YAML if both are supplied. -
--cpus TEXT
Number of vCPUs each instance must have (e.g.,--cpus=4(exactly 4) or--cpus=4+(at least 4)). This is used to automatically select the instance type. -
--memory TEXT
Amount of memory each instance must have in GB (e.g.,--memory=16(exactly 16GB),--memory=16+(at least 16GB)) -
--disk-size INTEGER
OS disk size in GBs. -
--image-id TEXT
Custom image id for launching the instances. Passing “none” resets the config. -
--env-file DOTENV_VALUES
Path to a dotenv file with environment variables to set on the remote node. If any values from--env-fileconflict with values set by--env, the--envvalue will be preferred. -
--env _PARSE_ENV_VAR
\ Environment variable to set on the remote node. It can be specified multiple times:-
--env MY_ENV=1: set$MY_ENVon the cluster to be 1. -
--env MY_ENV2=$HOME: set$MY_ENV2on the cluster to be the
$HOMEin the local environment where the CLI command is run.--env MY_ENV3: set$MY_ENV3on the cluster to be the
$MY_ENV3in the local environment. -
-
--gpus TEXT
Type and number of GPUs to use. Example values: “V100:8”, “V100” (short for a count of 1) If a new cluster is being launched by this command, this is the resources to provision. If an existing cluster is being reused, this is seen as the task demand, which must fit the cluster’s total resources and is used for scheduling the task. Overrides the “accelerators” config in the YAML if both are supplied. Passing “none” resets the config. -
--yes, -y
Skip confirmation prompt. -
--skip-image-check, -s
Skip Docker image validation checks for faster startup.
- If ENTRYPOINT points to a valid YAML file, it is read in as the task specification. Otherwise, it is interpreted as a bash command.
konduktor down
Tear down job(s).
Usage
JOBS...
--all, -a
Tear down all jobs.--all-users, --all_users
Include other users for teardown--yes, -y
Skip confirmation prompt.
- If both JOB and
--allare supplied, the latter takes precedence. - Tearing down a job will delete all associated containers (all billing stops), and any data on the containers disks will be lost. Accelerators (e.g., GPUs) that are part of the job will be deleted too.
- Wildcard patterns are supported using * characters. Ex: “test-” matches all jobs starting with “test-”, “-gpu” matches all jobs ending with “-gpu”.
konduktor stop
Suspend job(s) (manual/user-initiated).
Usage
JOBS...
--all, -a
Suspend all jobs.--all-users, --all_users
Include other users for suspension--yes, -y
Skip confirmation prompt.
- If both JOB and
--allare supplied, the latter takes precedence. - Suspending a job will pause execution and mark the job as SUSPENDED (by user). The job can be resumed later with
konduktor start. - If a job is suspended by the system (e.g., due to queueing), it will show as SUSPENDED (by system).
- Wildcard patterns are supported using * characters. Ex: “test-” matches all jobs starting with “test-”, “-gpu” matches all jobs ending with “-gpu”.
konduktor start
Resume suspended job(s) (manual/user-initiated).
Usage
JOBS...
--all, -a
Resume all suspended jobs.--all-users, --all_users
Include other users for resumption--yes, -y
Skip confirmation prompt.
- If both JOB and
--allare supplied, the latter takes precedence. - Resuming a job will restart execution from where it was suspended. Only suspended jobs can be resumed.
- This command works for both manually suspended jobs (SUSPENDED by user) and system-suspended jobs (SUSPENDED by system).
- Wildcard patterns are supported using * characters. Ex: “test-” matches all jobs starting with “test-”, “-gpu” matches all jobs ending with “-gpu”.
konduktor check
Check which clouds are available to use for storage with Konduktor
Usage
CLOUDS...
- This checks storage credentials for a cloud supported by konduktor. If a cloud is detected to be inaccessible, the reason and correction steps will be shown.
- If CLOUDS are specified, checks credentials for only those clouds.
- The enabled clouds are cached and form the “search space” to be considered for each task.
konduktor secret
Manage secrets used in Konduktor.
Usage
konduktor secret create
Create a new secret.
Usage
NAME
--kind CHOICE
Type of secret being created. More kinds coming soon. (default:default)--from-directory, --from_directory DIRECTORY
Path to a directory to store as a multi-file secret.--from-file, --from_file FILE
Path to a single file to store as a secret.--inline KEY=VALUE
Key=value pair to store as an env secret (only valid with —kind env).
konduktor secret delete
Delete a secret by name.
Usage
NAME
konduktor secret list
List secrets in the namespace.
Usage
--all-users, --all_users, -u
Show all secrets, including those not owned by the current user.
konduktor serve
Manage deployment serving with Konduktor.
Usage
konduktor serve launch
Launch a deployment to serve.
Usage
ENTRYPOINT...
-
--dryrun
If True, do not actually run the job. -
--detach-run, -d
If True, as soon as a job is submitted, return from this call and do not stream execution logs. -
--name, -n TEXT
Task name. Overrides the “name” config in the YAML if both are supplied. -
--workdir DIRECTORY
If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the “workdir” config in the YAML if both are supplied. -
--cloud TEXT
The cloud to use. If specified, overrides the “resources.cloud” config. Passing “none” resets the config. [defunct] currently only supports a single cloud -
--num-nodes INTEGER
Number of nodes to execute the task on. Overrides the “num_nodes” config in the YAML if both are supplied. -
--max-restarts INTEGER
Maximum number of jobset restarts allowed. Overrides YAML.Overrides the “max_restarts” config in the YAML if both are supplied. -
--completions INTEGER
Number of successful completions required. Overrides YAML.Overrides the “completions” config in the YAML if both are supplied. -
--cpus TEXT
Number of vCPUs each instance must have (e.g.,--cpus=4(exactly 4) or--cpus=4+(at least 4)). This is used to automatically select the instance type. -
--memory TEXT
Amount of memory each instance must have in GB (e.g.,--memory=16(exactly 16GB),--memory=16+(at least 16GB)) -
--disk-size INTEGER
OS disk size in GBs. -
--image-id TEXT
Custom image id for launching the instances. Passing “none” resets the config. -
--env-file DOTENV_VALUES
Path to a dotenv file with environment variables to set on the remote node. If any values from--env-fileconflict with values set by--env, the--envvalue will be preferred. -
--env _PARSE_ENV_VAR
\ Environment variable to set on the remote node. It can be specified multiple times:-
--env MY_ENV=1: set$MY_ENVon the cluster to be 1. -
--env MY_ENV2=$HOME: set$MY_ENV2on the cluster to be the
$HOMEin the local environment where the CLI command is run.--env MY_ENV3: set$MY_ENV3on the cluster to be the
$MY_ENV3in the local environment. -
-
--gpus TEXT
Type and number of GPUs to use. Example values: “V100:8”, “V100” (short for a count of 1) If a new cluster is being launched by this command, this is the resources to provision. If an existing cluster is being reused, this is seen as the task demand, which must fit the cluster’s total resources and is used for scheduling the task. Overrides the “accelerators” config in the YAML if both are supplied. Passing “none” resets the config. -
--min-replicas INTEGER
Minimum number of replicas to run for the service. Overrides the “min_replicas” field in the YAML if both are supplied. -
--max-replicas INTEGER
Maximum number of replicas to allow for the service. Overrides the “max_replicas” field in the YAML if both are supplied. -
--ports INTEGER
The container port on which your service will listen for HTTP traffic. Overrides the “ports” field in the YAML if both are supplied. -
--probe TEXT
The HTTP path to use for health checks (liveness, readiness, and startup probes). Overrides the “probe” field in the YAML if both are supplied. The service should respond with HTTP 200 on this path when healthy. -
--yes, -y
Skip confirmation prompt. -
--skip-image-check, -s
Skip Docker image validation checks for faster startup.
- If ENTRYPOINT points to a valid YAML file, it is read in as the task specification. Otherwise, it is interpreted as a bash command.
konduktor serve down
Tear down deployments (Deployment, Service, PodAutoscaler).
Usage
NAMES...
--all, -a
Tear down all deployments.--yes, -y
Skip confirmation prompt.
konduktor serve status
Show status of deployments launched via konduktor serve launch.
Usage
--all-users, -u
Show all deployments, including those not owned by the current user.--direct, -d
Force display of direct IP endpoints instead of trainy.us endpoints.