Managing Secrets

Konduktor makes it easy to manage sensitive credentials like API keys, SSH keys, cloud credentials, and environment variables through a built-in secret management system. Secrets are stored as Kubernetes Secrets and are automatically injected into the workloads of the user who created them. Some secrets (kind=env) can be exposed inside the workload as environment variables, while others (kind=default) are mounted as files or directories.

Creating Secrets

To create a secret, use the konduktor secret create command shown below. If a Konduktor secret with the same name already exists under your ownership, the new data will be merged with the existing contents rather than replacing it. This allows a one to many relationship between Konduktor secret names and secrets data (basically key:value pairs). If you need to replace a Konduktor secret entirely, delete the existing Konduktor secret first, then create a new one. Different users in the same cluster can create secrets with the same name without conflict as each user’s secrets are functionally isolated.

konduktor secret create [FLAGS] <SECRET_NAME>

Where valid [FLAGS] include:

--kind=<KIND> # see below for supported secret types

# mutually exclusive source flags:
--from-file=<FILE_PATH>
--from-directory=<DIRECTORY_PATH>
--inline KEY=VALUE

You can choose from several secret kinds, depending on how you want the secret to be injected into your workload:

1. kind=default

Use this kind to create general-purpose secrets that will be mounted directly into your pods as files or directories. Any of the three source flags are acceptable. Excluding the --kind flag will default to --kind=default.

$ konduktor secret create --kind=default --from-file=./tests/secrets/secret-file.txt my-default-file
$ konduktor secret create --kind=default --from-directory=./tests/secrets my-default-directory
$ konduktor secret create --kind=default --inline FOO=bar my-default-inline

They will be accessible within the workload at $KONDUKTOR_DEFAULT_SECRETS/<SECRET_NAME>/. Here is an example snippet of a workload yaml accessing the various default secrets:

name: test-default-secrets

run: |
  echo "Accessing file-based secret"
  ls "$KONDUKTOR_DEFAULT_SECRETS/my-default-file"
  cat "$KONDUKTOR_DEFAULT_SECRETS/my-default-file/secret-file.txt"
  
  # Note that directory name is preserved and whole directory (including subdirectories) is copied
  echo "Accessing directory-based secret"
  ls "$KONDUKTOR_DEFAULT_SECRETS/my-default-directory"
  ls "$KONDUKTOR_DEFAULT_SECRETS/my-default-directory/secrets/"
  cat "$KONDUKTOR_DEFAULT_SECRETS/my-default-directory/secrets/secret-file.txt"

  echo "Accessing inline secret"
  ls "$KONDUKTOR_DEFAULT_SECRETS/my-default-inline"
  cat "$KONDUKTOR_DEFAULT_SECRETS/my-default-inline/FOO"

2. kind=git-ssh

Use this kind to store an SSH private key for accessing Git repositories via SSH. You must use --from-file. A user is limited to only one git-ssh secret.

$ konduktor secret create --kind=git-ssh --from-file=~/.ssh/id_rsa my-ssh-name

This secret will be automatically mounted into workloads and configured via the environment variable GIT_SSH_COMMAND, allowing secure Git operations. Here is an example snippet of a workload yaml:

name: test-gitssh-secret

run: |
  git clone git@github.com:mygithubaccount/My-App.git

3. kind=env

Use this kind to store environment variables directly. Environment secrets are automatically injected into job containers as environment variables. You must use --inline to define key-value pairs directly in the CLI.

$ konduktor secret create --kind=env --inline FOO=bar my-env-name

Here is an example snippet of a workload yaml utilizing the automatic env secret mounting:

name: test-env-secret

run: |
  echo "My FOO is $FOO" # results in "My FOO is bar"

Validation when submitting a job

On installations with dependency admission enabled, submission checks required Secret and ConfigMap references before creating the workload. This covers Hub JobSets and Jobs, native Dynamo serving graphs, and Deployments on existing direct-cluster serving installations. The CLI, Python SDK, Hub serving requests, and direct Kubernetes submissions receive the same check. A missing required Secret, ConfigMap, or explicitly referenced key rejects the request. The error identifies the namespace, object name, and reference field without showing values. Create or repair the dependency in the selected namespace, then retry the same request. Rejected requests leave no workload object behind. References marked optional: true may be absent. Checks cover volumes, projections, envFrom, and key references in containers and init containers. ConfigMap file projections accept data and binaryData keys; environment key references require data. Dynamo graph-level environment references are checked where a component’s main container inherits them; explicit component environment overrides take precedence. Keep using konduktor secret create and the existing automatic injection rules; there is no separate task field listing required Secrets. Admission checks rendered references, so it cannot infer credentials mentioned only inside a shell script. Image pull credentials are excluded because public images may remain pullable without them. If a dependency disappears after admission, dependency synchronization still reports it while the job waits for dispatch. Cancellation and status updates remain available. Operators enable this check after upgrading the Hub controller image. See the rollout guide for activation and rollback.

Viewing Secrets

To view your own secrets:

$ konduktor secret list

To view secrets of all users in the cluster:

$ konduktor secret list --all-users
$ konduktor secret list -u

Deleting Secrets

To delete a secret, use the secret name. Users can only delete the secrets they created.

$ konduktor secret delete <SECRET_NAME>

There is a known issue where users who submit jobs from two different machines cannot manage/use the same set of secrets across workstations since ownership of the secret is determined currently by username + device MAC address. We are currently working to address this, but as a mitigation, users can recreate the same secrets on different workstations to get the same benefit.

Konduktor secrets are backed by Kubernetes secrets and as such have a size limit of 1MB. In practice, we recommend users only use secrets that are around 1K or less to prevent unexpected issues with secrets mounting

Get Started

CLI

User Guides

Cluster Operators

Creating Secrets

1. kind=default

2. kind=git-ssh

3. kind=env

Validation when submitting a job

Viewing Secrets

Deleting Secrets

​Creating Secrets

​1. kind=default

​2. kind=git-ssh

​3. kind=env

​Validation when submitting a job

​Viewing Secrets

​Deleting Secrets

Creating Secrets

1. kind=default

2. kind=git-ssh

3. kind=env

Validation when submitting a job

Viewing Secrets

Deleting Secrets