> ## Documentation Index > Fetch the complete documentation index at: https://docs.trainy.ai/llms.txt > Use this file to discover all available pages before exploring further. # CLI Reference > Reference for the Konduktor CLI **Using an AI coding assistant?** Install the [Konduktor Skills](https://github.com/Trainy-ai/konduktor-skills) plugin (`npx skills add Trainy-ai/konduktor-skills`) to let your AI construct CLI commands for you — run `/konduktor-cli` to get started. ### `konduktor status` Shows list of all the jobs with optional filtering and pagination. **Usage** ```bash theme={null} Usage: konduktor status [OPTIONS] ``` **Options** * `--all-users, -u`\ Show all jobs, including those not owned by the current user. * `--limit, -l INTEGER`\ Maximum number of jobs to display (e.g., --limit 100) * `--after TEXT`\ Show jobs created after this timestamp (e.g., --after "08/06/25 03:54PM", --after "08/06/25", --after "03:54PM") * `--before TEXT`\ Show jobs created before this timestamp (e.g., --before "08/06/25 03:54PM", --before "08/06/25", --before "03:54PM") **Examples** ```bash theme={null} konduktor status --limit 10 konduktor status --before "08/06/25 03:53PM" konduktor status --all-users --limit 10 --after "08/06/25 03:53PM" ``` **Notes** * When using --before or --after timestamps, "08/06/25" is equivalent to "08/06/25 00:00". * "03:53PM" is equivalent to "03:53:00PM". * Timestamps shown in "konduktor status" are truncated and are in the local timezone. Example: "03:53:55PM" → "03:53PM" would show up in --after "03:53PM" but not in --before "03:53PM". ### `konduktor logs` Retrieve/tail the log of a job. **Usage** ```bash theme={null} Usage: konduktor logs [OPTIONS] JOB_ID ``` **Arguments** * `JOB_ID` **Options** * `--status`\ \[DEPRECATED] If specified, do not show logs but exit with a status code for the job's status: 0 for succeeded, or 1 for all other statuses. * `--follow, --no-follow`\ Follow the logs of a job. If --no-follow is specified, print the log so far and exit. (default: --follow) * `--num-lines, --num_lines, -n INTEGER`\ The number of lines to display from the end of the log file. Default is -1 (no limit). *(default: `-1`)* * `--node-rank, --node_rank, -N INTEGER`\ The node rank to tail logs from. *(default: `0`)* * `--start-offset, --start_offset TEXT`\ Choose how much time from now to look back in logs. Examples: 30s, 5m, 2h, 1d. Default is 1h. Note: currently only applies when streaming (default --follow). With --no-follow, all available logs are returned. *(default: `1h`)* ### `konduktor launch` Launch a task. **Usage** ```bash theme={null} Usage: konduktor launch [OPTIONS] [ENTRYPOINT]... ``` **Arguments** * `ENTRYPOINT...` **Options** * `--dryrun`\ If True, do not actually run the job. * `--detach-run, -d`\ If True, as soon as a job is submitted, return from this call and do not stream execution logs. * `--name, -n TEXT`\ Task name. Overrides the "name" config in the YAML if both are supplied. * `--workdir DIRECTORY`\ If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the "workdir" config in the YAML if both are supplied. * `--cloud TEXT`\ The cloud to use. If specified, overrides the "resources.cloud" config. Passing "none" resets the config. \[defunct] currently only supports a single cloud * `--num-nodes INTEGER`\ Number of nodes to execute the task on. Overrides the "num\_nodes" config in the YAML if both are supplied. * `--max-restarts INTEGER`\ Maximum number of jobset restarts allowed. Overrides YAML.Overrides the "max\_restarts" config in the YAML if both are supplied. * `--completions INTEGER`\ Number of successful completions required. Overrides YAML.Overrides the "completions" config in the YAML if both are supplied. * `--cpus TEXT`\ Number of vCPUs each instance must have (e.g., `--cpus=4` (exactly 4) or `--cpus=4+` (at least 4)). This is used to automatically select the instance type. * `--memory TEXT`\ Amount of memory each instance must have in GB (e.g., `--memory=16` (exactly 16GB), `--memory=16+` (at least 16GB)) * `--disk-size INTEGER`\ OS disk size in GBs. * `--image-id TEXT`\ Custom image id for launching the instances. Passing "none" resets the config. * `--env-file DOTENV_VALUES`\ Path to a dotenv file with environment variables to set on the remote node. If any values from `--env-file` conflict with values set by `--env`, the `--env` value will be preferred. * `--env _PARSE_ENV_VAR`\ \\ Environment variable to set on the remote node. It can be specified multiple times: 1. `--env MY_ENV=1`: set `$MY_ENV` on the cluster to be 1. 2. `--env MY_ENV2=$HOME`: set `$MY_ENV2` on the cluster to be the same value of `$HOME` in the local environment where the CLI command is run. 3. `--env MY_ENV3`: set `$MY_ENV3` on the cluster to be the same value of `$MY_ENV3` in the local environment. * `--gpus TEXT`\ Type and number of GPUs to use. Example values: "V100:8", "V100" (short for a count of 1) If a new cluster is being launched by this command, this is the resources to provision. If an existing cluster is being reused, this is seen as the task demand, which must fit the cluster's total resources and is used for scheduling the task. Overrides the "accelerators" config in the YAML if both are supplied. Passing "none" resets the config. * `--yes, -y`\ Skip confirmation prompt. * `--skip-image-check, -s`\ Skip Docker image validation checks for faster startup. **Notes** * If ENTRYPOINT points to a valid YAML file, it is read in as the task specification. Otherwise, it is interpreted as a bash command. ### `konduktor down` Tear down job(s). **Usage** ```bash theme={null} Usage: konduktor down [OPTIONS] [JOBS]... ``` **Arguments** * `JOBS...` **Options** * `--all, -a`\ Tear down all jobs. * `--all-users, --all_users`\ Include other users for teardown * `--yes, -y`\ Skip confirmation prompt. **Examples** ```bash theme={null} # Tear down a specific job. konduktor down my_job # Tear down multiple jobs. konduktor down my_job1 my_job2 # Tear down all jobs matching a pattern. konduktor down "my_job-*" # Tear down all of this users jobs. konduktor down -a konduktor down --all # Tear down all jobs across all users konduktor down --all --all-users ``` **Notes** * If both JOB and `--all` are supplied, the latter takes precedence. * Tearing down a job will delete all associated containers (all billing stops), and any data on the containers disks will be lost. Accelerators (e.g., GPUs) that are part of the job will be deleted too. * Wildcard patterns are supported using \* characters. Ex: "test-*" matches all jobs starting with "test-", "*-gpu" matches all jobs ending with "-gpu". ### `konduktor stop` Suspend job(s) (manual/user-initiated). **Usage** ```bash theme={null} Usage: konduktor stop [OPTIONS] [JOBS]... ``` **Arguments** * `JOBS...` **Options** * `--all, -a`\ Suspend all jobs. * `--all-users, --all_users`\ Include other users for suspension * `--yes, -y`\ Skip confirmation prompt. **Examples** ```bash theme={null} # Suspend a specific job. konduktor stop my_job # Suspend multiple jobs. konduktor stop my_job1 my_job2 # Suspend all jobs matching a pattern. konduktor stop "my_job-*" # Suspend all of this users jobs. konduktor stop -a konduktor stop --all # Suspend all jobs across all users konduktor stop --all --all-users ``` **Notes** * If both JOB and `--all` are supplied, the latter takes precedence. * Suspending a job will pause execution and mark the job as SUSPENDED (by user). The job can be resumed later with `konduktor start`. * If a job is suspended by the system (e.g., due to queueing), it will show as SUSPENDED (by system). * Wildcard patterns are supported using \* characters. Ex: "test-*" matches all jobs starting with "test-", "*-gpu" matches all jobs ending with "-gpu". ### `konduktor start` Resume suspended job(s) (manual/user-initiated). **Usage** ```bash theme={null} Usage: konduktor start [OPTIONS] [JOBS]... ``` **Arguments** * `JOBS...` **Options** * `--all, -a`\ Resume all suspended jobs. * `--all-users, --all_users`\ Include other users for resumption * `--yes, -y`\ Skip confirmation prompt. **Examples** ```bash theme={null} # Resume a specific job. konduktor start my_job # Resume multiple jobs. konduktor start my_job1 my_job2 # Resume all jobs matching a pattern. konduktor start "my_job-*" # Resume all of this users suspended jobs. konduktor start -a konduktor start --all # Resume all suspended jobs across all users konduktor start --all --all-users ``` **Notes** * If both JOB and `--all` are supplied, the latter takes precedence. * Resuming a job will restart execution from where it was suspended. Only suspended jobs can be resumed. * This command works for both manually suspended jobs (SUSPENDED by user) and system-suspended jobs (SUSPENDED by system). * Wildcard patterns are supported using \* characters. Ex: "test-*" matches all jobs starting with "test-", "*-gpu" matches all jobs ending with "-gpu". ### `konduktor check` Check which clouds are available to use for storage with Konduktor **Usage** ```bash theme={null} Usage: konduktor check [OPTIONS] CLOUDS... ``` **Arguments** * `CLOUDS...` **Examples** ```bash theme={null} # Check only specific clouds - gs, s3. konduktor check gs konduktor check s3 ``` **Notes** * This checks storage credentials for a cloud supported by konduktor. If a cloud is detected to be inaccessible, the reason and correction steps will be shown. * If CLOUDS are specified, checks credentials for only those clouds. * The enabled clouds are cached and form the "search space" to be considered for each task. ## `konduktor secret` Manage secrets used in Konduktor. **Usage** ```bash theme={null} konduktor secret COMMAND ``` **Examples** ```bash theme={null} konduktor secret create --kind git-ssh --from-file ~/.ssh/id_rsa my-ssh-name konduktor secret create --kind env --inline FOO=bar my-env-name konduktor secret delete my-ssh-name konduktor secret list ``` ### `konduktor secret create` Create a new secret. **Usage** ```bash theme={null} Usage: konduktor secret create [OPTIONS] NAME ``` **Arguments** * `NAME` **Options** * `--kind CHOICE`\ Type of secret being created. More kinds coming soon. *(default: `default`)* * `--from-directory, --from_directory DIRECTORY`\ Path to a directory to store as a multi-file secret. * `--from-file, --from_file FILE`\ Path to a single file to store as a secret. * `--inline KEY=VALUE`\ Key=value pair to store as an env secret (only valid with --kind env). ### `konduktor secret delete` Delete a secret by name. **Usage** ```bash theme={null} Usage: konduktor secret delete [OPTIONS] NAME ``` **Arguments** * `NAME` ### `konduktor secret list` List secrets in the namespace. **Usage** ```bash theme={null} Usage: konduktor secret list [OPTIONS] ``` **Options** * `--all-users, --all_users, -u`\ Show all secrets, including those not owned by the current user. ## `konduktor serve` Manage deployment serving with Konduktor. **Usage** ```bash theme={null} konduktor serve COMMAND ``` **Examples** ```bash theme={null} konduktor serve launch my-deployment konduktor serve down my-deployment konduktor serve status ``` ### `konduktor serve launch` Launch a deployment to serve. **Usage** ```bash theme={null} Usage: konduktor serve launch [OPTIONS] [ENTRYPOINT]... ``` **Arguments** * `ENTRYPOINT...` **Options** * `--dryrun`\ If True, do not actually run the job. * `--detach-run, -d`\ If True, as soon as a job is submitted, return from this call and do not stream execution logs. * `--name, -n TEXT`\ Task name. Overrides the "name" config in the YAML if both are supplied. * `--workdir DIRECTORY`\ If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the "workdir" config in the YAML if both are supplied. * `--cloud TEXT`\ The cloud to use. If specified, overrides the "resources.cloud" config. Passing "none" resets the config. \[defunct] currently only supports a single cloud * `--num-nodes INTEGER`\ Number of nodes to execute the task on. Overrides the "num\_nodes" config in the YAML if both are supplied. * `--max-restarts INTEGER`\ Maximum number of jobset restarts allowed. Overrides YAML.Overrides the "max\_restarts" config in the YAML if both are supplied. * `--completions INTEGER`\ Number of successful completions required. Overrides YAML.Overrides the "completions" config in the YAML if both are supplied. * `--cpus TEXT`\ Number of vCPUs each instance must have (e.g., `--cpus=4` (exactly 4) or `--cpus=4+` (at least 4)). This is used to automatically select the instance type. * `--memory TEXT`\ Amount of memory each instance must have in GB (e.g., `--memory=16` (exactly 16GB), `--memory=16+` (at least 16GB)) * `--disk-size INTEGER`\ OS disk size in GBs. * `--image-id TEXT`\ Custom image id for launching the instances. Passing "none" resets the config. * `--env-file DOTENV_VALUES`\ Path to a dotenv file with environment variables to set on the remote node. If any values from `--env-file` conflict with values set by `--env`, the `--env` value will be preferred. * `--env _PARSE_ENV_VAR`\ \\ Environment variable to set on the remote node. It can be specified multiple times: 1. `--env MY_ENV=1`: set `$MY_ENV` on the cluster to be 1. 2. `--env MY_ENV2=$HOME`: set `$MY_ENV2` on the cluster to be the same value of `$HOME` in the local environment where the CLI command is run. 3. `--env MY_ENV3`: set `$MY_ENV3` on the cluster to be the same value of `$MY_ENV3` in the local environment. * `--gpus TEXT`\ Type and number of GPUs to use. Example values: "V100:8", "V100" (short for a count of 1) If a new cluster is being launched by this command, this is the resources to provision. If an existing cluster is being reused, this is seen as the task demand, which must fit the cluster's total resources and is used for scheduling the task. Overrides the "accelerators" config in the YAML if both are supplied. Passing "none" resets the config. * `--min-replicas INTEGER`\ Minimum number of replicas to run for the service. Overrides the "min\_replicas" field in the YAML if both are supplied. * `--max-replicas INTEGER`\ Maximum number of replicas to allow for the service. Overrides the "max\_replicas" field in the YAML if both are supplied. * `--ports INTEGER`\ The container port on which your service will listen for HTTP traffic. Overrides the "ports" field in the YAML if both are supplied. * `--probe TEXT`\ The HTTP path to use for health checks (liveness, readiness, and startup probes). Overrides the "probe" field in the YAML if both are supplied. The service should respond with HTTP 200 on this path when healthy. * `--yes, -y`\ Skip confirmation prompt. * `--skip-image-check, -s`\ Skip Docker image validation checks for faster startup. **Notes** * If ENTRYPOINT points to a valid YAML file, it is read in as the task specification. Otherwise, it is interpreted as a bash command. ### `konduktor serve down` Tear down deployments (Deployment, Service, PodAutoscaler). **Usage** ```bash theme={null} Usage: konduktor serve down [OPTIONS] [NAMES]... ``` **Arguments** * `NAMES...` **Options** * `--all, -a`\ Tear down all deployments. * `--yes, -y`\ Skip confirmation prompt. **Examples** ```bash theme={null} konduktor serve down my-deployment konduktor serve down -a ``` ### `konduktor serve status` Show status of deployments launched via `konduktor serve launch`. **Usage** ```bash theme={null} Usage: konduktor serve status [OPTIONS] ``` **Options** * `--all-users, -u`\ Show all deployments, including those not owned by the current user. * `--direct, -d`\ Force display of direct IP endpoints instead of trainy.us endpoints.