Skip to main content

konduktor status

Shows list of all the jobs with optional filtering and pagination. Usage
Usage: konduktor status [OPTIONS]
Options
  • --all-users, -u
    Show all jobs, including those not owned by the current user.
  • --limit, -l INTEGER
    Maximum number of jobs to display (e.g., —limit 100)
  • --after TEXT
    Show jobs created after this timestamp (e.g., —after “08/06/25 03:54PM”, —after “08/06/25”, —after “03:54PM”)
  • --before TEXT
    Show jobs created before this timestamp (e.g., —before “08/06/25 03:54PM”, —before “08/06/25”, —before “03:54PM”)
Examples
konduktor status --limit 10
konduktor status --before "08/06/25 03:53PM"
konduktor status --all-users --limit 10 --after "08/06/25 03:53PM"
Notes
  • When using —before or —after timestamps, “08/06/25” is equivalent to “08/06/25 00:00”.
  • “03:53PM” is equivalent to “03:53:00PM”.
  • Timestamps shown in “konduktor status” are truncated and are in the local timezone. Example: “03:53:55PM” → “03:53PM” would show up in —after “03:53PM” but not in —before “03:53PM”.

konduktor logs

Retrieve/tail the log of a job. Usage
Usage: konduktor logs [OPTIONS] JOB_ID
Arguments
  • JOB_ID
Options
  • --status
    [DEPRECATED] If specified, do not show logs but exit with a status code for the job’s status: 0 for succeeded, or 1 for all other statuses.
  • --follow, --no-follow
    Follow the logs of a job. If —no-follow is specified, print the log so far and exit. (default: —follow)
  • --num-lines, --num_lines, -n INTEGER
    The number of lines to display from the end of the log file. Default is -1 (no limit). (default: -1)
  • --node-rank, --node_rank, -N INTEGER
    The node rank to tail logs from. (default: 0)
  • --start-offset, --start_offset TEXT
    Choose how much time from now to look back in logs. Examples: 30s, 5m, 2h, 1d. Default is 1h. Note: currently only applies when streaming (default —follow). With —no-follow, all available logs are returned. (default: 1h)

konduktor launch

Launch a task. Usage
Usage: konduktor launch [OPTIONS] [ENTRYPOINT]...
Arguments
  • ENTRYPOINT...
Options
  • --dryrun
    If True, do not actually run the job.
  • --detach-run, -d
    If True, as soon as a job is submitted, return from this call and do not stream execution logs.
  • --name, -n TEXT
    Task name. Overrides the “name” config in the YAML if both are supplied.
  • --workdir DIRECTORY
    If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the “workdir” config in the YAML if both are supplied.
  • --cloud TEXT
    The cloud to use. If specified, overrides the “resources.cloud” config. Passing “none” resets the config. [defunct] currently only supports a single cloud
  • --num-nodes INTEGER
    Number of nodes to execute the task on. Overrides the “num_nodes” config in the YAML if both are supplied.
  • --max-restarts INTEGER
    Maximum number of jobset restarts allowed. Overrides YAML.Overrides the “max_restarts” config in the YAML if both are supplied.
  • --completions INTEGER
    Number of successful completions required. Overrides YAML.Overrides the “completions” config in the YAML if both are supplied.
  • --cpus TEXT
    Number of vCPUs each instance must have (e.g., --cpus=4 (exactly 4) or --cpus=4+ (at least 4)). This is used to automatically select the instance type.
  • --memory TEXT
    Amount of memory each instance must have in GB (e.g., --memory=16 (exactly 16GB), --memory=16+ (at least 16GB))
  • --disk-size INTEGER
    OS disk size in GBs.
  • --image-id TEXT
    Custom image id for launching the instances. Passing “none” resets the config.
  • --env-file DOTENV_VALUES
    Path to a dotenv file with environment variables to set on the remote node. If any values from --env-file conflict with values set by --env, the --env value will be preferred.
  • --env _PARSE_ENV_VAR
    \
    Environment variable to set on the remote node. It can be specified multiple times:
    1. --env MY_ENV=1: set $MY_ENV on the cluster to be 1.
    2. --env MY_ENV2=$HOME: set $MY_ENV2 on the cluster to be the
    same value of $HOME in the local environment where the CLI command is run.
    1. --env MY_ENV3: set $MY_ENV3 on the cluster to be the
    same value of $MY_ENV3 in the local environment.
  • --gpus TEXT
    Type and number of GPUs to use. Example values: “V100:8”, “V100” (short for a count of 1) If a new cluster is being launched by this command, this is the resources to provision. If an existing cluster is being reused, this is seen as the task demand, which must fit the cluster’s total resources and is used for scheduling the task. Overrides the “accelerators” config in the YAML if both are supplied. Passing “none” resets the config.
  • --yes, -y
    Skip confirmation prompt.
  • --skip-image-check, -s
    Skip Docker image validation checks for faster startup.
Notes
  • If ENTRYPOINT points to a valid YAML file, it is read in as the task specification. Otherwise, it is interpreted as a bash command.

konduktor down

Tear down job(s). Usage
Usage: konduktor down [OPTIONS] [JOBS]...
Arguments
  • JOBS...
Options
  • --all, -a
    Tear down all jobs.
  • --all-users, --all_users
    Include other users for teardown
  • --yes, -y
    Skip confirmation prompt.
Examples
# Tear down a specific job.
konduktor down my_job

# Tear down multiple jobs.
konduktor down my_job1 my_job2

# Tear down all jobs matching a pattern.
konduktor down "my_job-*"

# Tear down all of this users jobs.
konduktor down -a
konduktor down --all

# Tear down all jobs across all users
konduktor down --all --all-users
Notes
  • If both JOB and --all are supplied, the latter takes precedence.
  • Tearing down a job will delete all associated containers (all billing stops), and any data on the containers disks will be lost. Accelerators (e.g., GPUs) that are part of the job will be deleted too.
  • Wildcard patterns are supported using * characters. Ex: “test-” matches all jobs starting with “test-”, “-gpu” matches all jobs ending with “-gpu”.

konduktor stop

Suspend job(s) (manual/user-initiated). Usage
Usage: konduktor stop [OPTIONS] [JOBS]...
Arguments
  • JOBS...
Options
  • --all, -a
    Suspend all jobs.
  • --all-users, --all_users
    Include other users for suspension
  • --yes, -y
    Skip confirmation prompt.
Examples
# Suspend a specific job.
konduktor stop my_job

# Suspend multiple jobs.
konduktor stop my_job1 my_job2

# Suspend all jobs matching a pattern.
konduktor stop "my_job-*"

# Suspend all of this users jobs.
konduktor stop -a
konduktor stop --all

# Suspend all jobs across all users
konduktor stop --all --all-users
Notes
  • If both JOB and --all are supplied, the latter takes precedence.
  • Suspending a job will pause execution and mark the job as SUSPENDED (by user). The job can be resumed later with konduktor start.
  • If a job is suspended by the system (e.g., due to queueing), it will show as SUSPENDED (by system).
  • Wildcard patterns are supported using * characters. Ex: “test-” matches all jobs starting with “test-”, “-gpu” matches all jobs ending with “-gpu”.

konduktor start

Resume suspended job(s) (manual/user-initiated). Usage
Usage: konduktor start [OPTIONS] [JOBS]...
Arguments
  • JOBS...
Options
  • --all, -a
    Resume all suspended jobs.
  • --all-users, --all_users
    Include other users for resumption
  • --yes, -y
    Skip confirmation prompt.
Examples
# Resume a specific job.
konduktor start my_job

# Resume multiple jobs.
konduktor start my_job1 my_job2

# Resume all jobs matching a pattern.
konduktor start "my_job-*"

# Resume all of this users suspended jobs.
konduktor start -a
konduktor start --all

# Resume all suspended jobs across all users
konduktor start --all --all-users
Notes
  • If both JOB and --all are supplied, the latter takes precedence.
  • Resuming a job will restart execution from where it was suspended. Only suspended jobs can be resumed.
  • This command works for both manually suspended jobs (SUSPENDED by user) and system-suspended jobs (SUSPENDED by system).
  • Wildcard patterns are supported using * characters. Ex: “test-” matches all jobs starting with “test-”, “-gpu” matches all jobs ending with “-gpu”.

konduktor check

Check which clouds are available to use for storage with Konduktor Usage
Usage: konduktor check [OPTIONS] CLOUDS...
Arguments
  • CLOUDS...
Examples
# Check only specific clouds - gs, s3.
konduktor check gs
konduktor check s3
Notes
  • This checks storage credentials for a cloud supported by konduktor. If a cloud is detected to be inaccessible, the reason and correction steps will be shown.
  • If CLOUDS are specified, checks credentials for only those clouds.
  • The enabled clouds are cached and form the “search space” to be considered for each task.

konduktor secret

Manage secrets used in Konduktor. Usage
konduktor secret COMMAND
Examples
konduktor secret create --kind git-ssh --from-file ~/.ssh/id_rsa my-ssh-name
konduktor secret create --kind env --inline FOO=bar my-env-name
konduktor secret delete my-ssh-name
konduktor secret list

konduktor secret create

Create a new secret. Usage
Usage: konduktor secret create [OPTIONS] NAME
Arguments
  • NAME
Options
  • --kind CHOICE
    Type of secret being created. More kinds coming soon. (default: default)
  • --from-directory, --from_directory DIRECTORY
    Path to a directory to store as a multi-file secret.
  • --from-file, --from_file FILE
    Path to a single file to store as a secret.
  • --inline KEY=VALUE
    Key=value pair to store as an env secret (only valid with —kind env).

konduktor secret delete

Delete a secret by name. Usage
Usage: konduktor secret delete [OPTIONS] NAME
Arguments
  • NAME

konduktor secret list

List secrets in the namespace. Usage
Usage: konduktor secret list [OPTIONS]
Options
  • --all-users, --all_users, -u
    Show all secrets, including those not owned by the current user.

konduktor serve

Manage deployment serving with Konduktor. Usage
konduktor serve COMMAND
Examples
konduktor serve launch my-deployment
konduktor serve down my-deployment
konduktor serve status

konduktor serve launch

Launch a deployment to serve. Usage
Usage: konduktor serve launch [OPTIONS] [ENTRYPOINT]...
Arguments
  • ENTRYPOINT...
Options
  • --dryrun
    If True, do not actually run the job.
  • --detach-run, -d
    If True, as soon as a job is submitted, return from this call and do not stream execution logs.
  • --name, -n TEXT
    Task name. Overrides the “name” config in the YAML if both are supplied.
  • --workdir DIRECTORY
    If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the “workdir” config in the YAML if both are supplied.
  • --cloud TEXT
    The cloud to use. If specified, overrides the “resources.cloud” config. Passing “none” resets the config. [defunct] currently only supports a single cloud
  • --num-nodes INTEGER
    Number of nodes to execute the task on. Overrides the “num_nodes” config in the YAML if both are supplied.
  • --max-restarts INTEGER
    Maximum number of jobset restarts allowed. Overrides YAML.Overrides the “max_restarts” config in the YAML if both are supplied.
  • --completions INTEGER
    Number of successful completions required. Overrides YAML.Overrides the “completions” config in the YAML if both are supplied.
  • --cpus TEXT
    Number of vCPUs each instance must have (e.g., --cpus=4 (exactly 4) or --cpus=4+ (at least 4)). This is used to automatically select the instance type.
  • --memory TEXT
    Amount of memory each instance must have in GB (e.g., --memory=16 (exactly 16GB), --memory=16+ (at least 16GB))
  • --disk-size INTEGER
    OS disk size in GBs.
  • --image-id TEXT
    Custom image id for launching the instances. Passing “none” resets the config.
  • --env-file DOTENV_VALUES
    Path to a dotenv file with environment variables to set on the remote node. If any values from --env-file conflict with values set by --env, the --env value will be preferred.
  • --env _PARSE_ENV_VAR
    \
    Environment variable to set on the remote node. It can be specified multiple times:
    1. --env MY_ENV=1: set $MY_ENV on the cluster to be 1.
    2. --env MY_ENV2=$HOME: set $MY_ENV2 on the cluster to be the
    same value of $HOME in the local environment where the CLI command is run.
    1. --env MY_ENV3: set $MY_ENV3 on the cluster to be the
    same value of $MY_ENV3 in the local environment.
  • --gpus TEXT
    Type and number of GPUs to use. Example values: “V100:8”, “V100” (short for a count of 1) If a new cluster is being launched by this command, this is the resources to provision. If an existing cluster is being reused, this is seen as the task demand, which must fit the cluster’s total resources and is used for scheduling the task. Overrides the “accelerators” config in the YAML if both are supplied. Passing “none” resets the config.
  • --min-replicas INTEGER
    Minimum number of replicas to run for the service. Overrides the “min_replicas” field in the YAML if both are supplied.
  • --max-replicas INTEGER
    Maximum number of replicas to allow for the service. Overrides the “max_replicas” field in the YAML if both are supplied.
  • --ports INTEGER
    The container port on which your service will listen for HTTP traffic. Overrides the “ports” field in the YAML if both are supplied.
  • --probe TEXT
    The HTTP path to use for health checks (liveness, readiness, and startup probes). Overrides the “probe” field in the YAML if both are supplied. The service should respond with HTTP 200 on this path when healthy.
  • --yes, -y
    Skip confirmation prompt.
  • --skip-image-check, -s
    Skip Docker image validation checks for faster startup.
Notes
  • If ENTRYPOINT points to a valid YAML file, it is read in as the task specification. Otherwise, it is interpreted as a bash command.

konduktor serve down

Tear down deployments (Deployment, Service, PodAutoscaler). Usage
Usage: konduktor serve down [OPTIONS] [NAMES]...
Arguments
  • NAMES...
Options
  • --all, -a
    Tear down all deployments.
  • --yes, -y
    Skip confirmation prompt.
Examples
konduktor serve down my-deployment
konduktor serve down -a

konduktor serve status

Show status of deployments launched via konduktor serve launch. Usage
Usage: konduktor serve status [OPTIONS]
Options
  • --all-users, -u
    Show all deployments, including those not owned by the current user.
  • --direct, -d
    Force display of direct IP endpoints instead of trainy.us endpoints.