Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.trainy.ai/llms.txt

Use this file to discover all available pages before exploring further.

These steps should be completed after receiving access to a Trainy k8s cluster via a kubeconfig file after which you can inspect your cluster using k8s tooling like kubectl and through the konduktor CLI tool. For instructions on how to get access to your cluster see Authentication.

​
Installation

We maintain a nightly build of our latest client on PyPi konduktor-nightly 
$ pip install konduktor-nightly

​
(Optional) Setup Cloud Storage Credentials

We currently support the following cloud object stores for syncing files —> workloads with more incoming!
  • Google Cloud Storage gs - Supported ✅
  • Amazon S3 s3 - Supported ✅
  • Cloudflare R2 r2 - Coming soon 🚧

​
Google Cloud Storage Requirements

  1. Install with pip install konduktor-nightly
  2. Check that your GCP user or service account has the minimal GCP permissions.
  3. Ensure that ~/.konduktor/config.yaml is configured correctly for GS:
allowed_clouds:
  - gs
  1. Use local credentials OR set env variables: Coming soon 🚧
  2. Confirm your GS credentials and setup with:
$ konduktor check gs

​
Amazon S3 Requirements

  1. Install with pip install konduktor-nightly[s3]
  2. Check that your AWS user has the minimal AWS user permissions.
  3. Ensure that ~/.konduktor/config.yaml is configured correctly for S3:
allowed_clouds:
  - s3
  1. Use local ~/.aws/ credentials OR set env variables:
~/.aws/config 
[default]
region = us-east-1
~/.aws/credentials 
[default]
aws_access_key_id=ASIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
OR 
$ export AWS_ACCESS_KEY_ID="ASIAIOSFODNN7EXAMPLE"
$ export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
$ export AWS_DEFAULT_REGION="us-east-1"
  1. Confirm your S3 credentials and setup with:
$ konduktor check s3

​
(Optional) Setup Private Container Registry Credentials

To have your Trainy jobs use images from private container registries (k8s docs), you must create one or more Kubernetes docker-registry secrets containing your registry credentials. Each secret stores the authentication needed for a specific registry (Docker Hub or GCR). Konduktor will use these secrets when pulling your job images. After creating the secret(s), reference them in your Konduktor config (~/.konduktor/config.yaml) so they are automatically attached to your pods. 
kubernetes:
  pod_config:
    spec:
      imagePullSecrets:
        - name: regcred-dockerhub
        - name: regcred-gcr
The names in imagePullSecrets must match the secrets you create below.

​
Create a Docker Hub Registry Secret

  1. Create your secret
$ kubectl create secret docker-registry regcred-dockerhub \
    --docker-server=https://index.docker.io/v1/ \
    --docker-username=<your-name> \
    --docker-password=<your-pword> \
    --docker-email=<your-email>
  1. Verify that the secret was created
$ kubectl get secrets
NAME                      TYPE                              DATA   AGE
regcred-dockerhub         kubernetes.io/dockerconfigjson    1      67s

​
Create a Google Container / Artifact Registry Secret

  1. Create a GCP Service Account for registry access
$ gcloud iam service-accounts create gcr-access --display-name "GCR Access"
  1. Grant read permissions for container images
# For GCR (legacy)
$ gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member "serviceAccount:gcr-access@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
    --role "roles/storage.objectViewer"

# For Artifact Registry (recommended)
$ gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member "serviceAccount:gcr-access@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
    --role "roles/artifactregistry.reader"
  1. Generate a JSON key for that service account
$ gcloud iam service-accounts keys create gcr-key.json --iam-account gcr-access@YOUR_PROJECT_ID.iam.gserviceaccount.com
  1. Create your secret
# For GCR (legacy)
$ kubectl create secret docker-registry regcred-gcr \
    --docker-server=gcr.io \
    --docker-username=_json_key \
    --docker-password="$(cat ~/gcr-key.json)" \
    --docker-email=<your-email> # CHANGE THIS LINE

# For Artifact Registry (recommended)
$ kubectl create secret docker-registry regcred-gcr \
    --docker-server=us-central1-docker.pkg.dev \
    --docker-username=_json_key \
    --docker-password="$(cat ~/gcr-key.json)" \
    --docker-email=<your-email> # CHANGE THIS LINE
  1. Verify that the secret was created
$ kubectl get secrets
NAME                TYPE                              DATA   AGE
regcred-gcr         kubernetes.io/dockerconfigjson    1      67s