Skip to main content
Konduktor automatically provides a rich set of environment variables in your containers that give you access to job metadata, networking information, and system configuration. These variables are automatically set when your container starts and can be accessed using standard shell syntax.

​
Setting Environment Variables

Environment variables in Konduktor can come from multiple sources, depending on how your workload and configuration are defined. When the same variable name is defined in more than one place, Konduktor applies a priority order to decide which value takes precedence.

​
Priority Order

If the same variable appears in multiple places, the highest priority source wins. Priorities from highest to lowest:
PrioritySourceDescription
1CLIVariables passed directly via the CLI with konduktor launch
2Task (task.yaml)Variables defined under the envs: block of a task.yaml
3Environment Secrets (kind=env)Variables created using konduktor secret create --kind=env
4Config (~/.konduktor/config.yaml)Variables defined globally in the Konduktor configuration file at ~/.konduktor/config.yaml
5Other/System DefaultsVariables automatically generated by Konduktor (ex. KONDUKTOR_JOB_NAME, NUM_NODES, etc.).
While the global priority is CLI > task.yaml > env secret > config.yaml > system, note that any exports (ex. export FOO=bar) done inside your run: script happens at runtime and will override the effective value for all subsequent commands in that script. So basically runtime > everything.

​
Examples

​
CLI

$ konduktor launch --env FOO=bar task.yaml

$ export FOO="bar" && konduktor launch --env FOO task.yaml

​
task.yaml

envs:
  FOO: bar

​
Env Secret

$ konduktor secret create --kind=env --inline FOO=bar my-env-secret-name

​
config.yaml

kubernetes:
  pod_config:
    spec:
      containers:
        - name: konduktor-container
          env:
            - name: FOO
              value: bar

​
Accessing Environment Variables

Environment variables can be accessed in several ways:

​
Shell Scripts

​
Task.yaml:

run: |
    # List all env vars
    env

    # Using $VAR syntax (recommended)
    echo "Job name: $KONDUKTOR_JOB_NAME"
    echo "Number of nodes: $NUM_NODES"

    # Using ${VAR} syntax (useful for concatenation)
    echo "Master address: ${MASTER_ADDR}"
    echo "Local IP: ${LOCAL_ADDR}"

    # Using ${VAR:-default} for fallback values
    echo "GPU count: ${NUM_GPUS_PER_NODE:-0}"

​
Python

​
Task.yaml:

run: |
    python pythonfile.py

​
pythonfile.py:

import os

# Get environment variables
job_name = os.environ.get('KONDUKTOR_JOB_NAME', 'unknown')
num_nodes = int(os.environ.get('NUM_NODES', '1'))
gpu_count = int(os.environ.get('NUM_GPUS_PER_NODE', '0'))

print(f"Running job: {job_name}")
print(f"Nodes: {num_nodes}")
print(f"GPUs per node: {gpu_count}")

​
Node.js

​
Task.yaml:

run: |
    node jsfile.js

​
jsfile.js:

// Get environment variables
const jobName = process.env.KONDUKTOR_JOB_NAME || 'unknown';
const numNodes = parseInt(process.env.NUM_NODES) || 1;
const gpuCount = parseInt(process.env.NUM_GPUS_PER_NODE) || 0;

console.log(`Running job: ${jobName}`);
console.log(`Nodes: ${numNodes}`);
console.log(`GPUs per node: ${gpuCount}`);

​
Core Konduktor Environment Variables

These variables are always available in every Konduktor container:

​
Job Information

VariableDescriptionExample Value
KONDUKTOR_JOB_NAMEUnique identifier for the current jobmy-training-job-a1b2
NUM_NODESTotal number of nodes in the job2
NUM_GPUS_PER_NODENumber of GPUs allocated per node8

​
Networking Information

VariableDescriptionExample Value
NODE_HOST_IPSComma-separated list of all node hostnamesjob-123-workers-0-0.job-123,job-123-workers-0-1.job-123
MASTER_ADDRHostname of the master/rank 0 nodejob-123-workers-0-0.job-123
LOCAL_ADDRPod’s internal IP address10.104.2.17
RANKCurrent node’s rank (0 for master, 1+ for workers)0

​
System Configuration

VariableDescriptionExample Value
PYTHONUNBUFFEREDPython output buffering setting0
JOB_COMPLETION_INDEXKubernetes job completion index0
RESTART_ATTEMPTJob restart attempt number (0 for first attempt, 1+ for retries)0

​
Conditional Environment Variables

These variables are only set when specific features are enabled:

​
Tailscale (when tailscale.secret_name is configured)

VariableDescriptionExample Value
TS_USERSPACETailscale userspace networking flagtrue
TS_AUTHKEYTailscale authentication keytskey-auth-...
POD_NAMEKubernetes pod namejob-123-workers-0-0-abc123
POD_UIDKubernetes pod UIDa1b2c3d4-e5f6-7890-abcd-ef1234567890

​
SSH (when ssh.enable is true)

VariableDescriptionExample Value
KONDUKTOR_SSHPUBPublic SSH key for the jobssh-rsa AAAAB3NzaC1...
KONDUKTOR_SSHPRIVPrivate SSH key for the job-----BEGIN OPENSSH PRIVATE KEY-----
KONDUKTOR_SSH_PORTSSH port number2222

​
Git SSH (when git-ssh secret exists)

VariableDescriptionValue
GIT_SSH_COMMANDSSH command for Git operationsssh -i /run/konduktor/git-ssh-secret/gitkey -o StrictHostKeyChecking=no

​
Default Secrets (when default secrets exist)

VariableDescriptionValue
KONDUKTOR_DEFAULT_SECRETSPath to mounted default secrets/konduktor/default-secrets

​
Practical Examples

​
WANDB Integration

import os
import wandb

# Automatically name your WANDB run after the Konduktor job
job_name = os.environ['KONDUKTOR_JOB_NAME']
num_nodes = int(os.environ['NUM_NODES'])
gpu_count = int(os.environ['NUM_GPUS_PER_NODE'])

wandb.init(
    project="my-training-project",
    name=job_name,
    config={
        "nodes": num_nodes,
        "gpus_per_node": gpu_count,
        "total_gpus": num_nodes * gpu_count
    }
)

print(f"WANDB run started: {job_name}")

​
Troubleshooting

If an environment variable is not set as expected:
  1. Check if the feature is enabled (e.g., SSH, Tailscale)
  2. Verify the variable name spelling
  3. Check the Konduktor logs for any errors
  4. Ensure you’re running the latest version of Konduktor