> ## Documentation Index > Fetch the complete documentation index at: https://docs.trainy.ai/llms.txt > Use this file to discover all available pages before exploring further. # Install Cluster Telemetry > Trainy: connect your own GPU cluster to Trainy in one command — metrics, infrastructure logs, Kubernetes events and GPU node health checks, installed with Helm and shipped from a namespace you control. This guide is for the team that operates the Kubernetes cluster. It installs the components Trainy needs to see your cluster's health, so we can run dashboards, alerts and GPU fault triage against it. The whole install is a handful of Helm releases and one custom resource. It adds no Prometheus, no Grafana, no exporters and no scheduler, and it does not modify the monitoring stack you already run. Before you start you need two things from Trainy: a **cluster identifier** (everything you ship is filed under it) and **confirmation that your cluster's egress IP is allowlisted** on the ingest endpoint. Without the allowlist the install comes up perfectly healthy and ships nothing. ## What gets installed | Component | What it is | What it does | | ------------------------ | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | VictoriaMetrics operator | One Deployment | Converts the ServiceMonitors your monitoring stack already defines into scrape targets for the agent below. Installs no Prometheus and no storage. | | Metrics shipper | A `VMAgent` custom resource | Scrapes those targets and remote-writes a **copy** to Trainy. Your Prometheus keeps scraping exactly what it scraped before. | | Log shipper | OpenTelemetry Collector, DaemonSet | Ships pod logs from an **allowlist of infrastructure namespaces**. Default deny. | | Event shipper | OpenTelemetry Collector, Deployment | Ships Kubernetes events (OOMKills, scheduling failures, node pressure) from allowlisted namespaces, plus all node events. | | Node health | `trainy-npd` DaemonSet + remediation controller | Runs GPU, RDMA, PCIe, NVMe and kernel checks, publishing results as `trainy.ai/*` NodeConditions, and acts on them by default — see [Node health and auto-remediation](#node-health-and-auto-remediation). | | Kernel logs | `dmesg`, DaemonSet on every node | Streams each node's kernel ring buffer, which the log shipper forwards. Runs **privileged** — see below. | **Exporters are not installed.** `kube-state-metrics`, `node-exporter` and the GPU metrics exporter belong to whoever runs your monitoring stack. This install reads them and reports at preflight which ones it can find. ## What leaves your cluster Worth reading before you install, and worth showing to whoever asks. The agent discovers targets by converting the ServiceMonitors already in your cluster, plus one static job for the GPU exporter. That is what makes the install work without us writing scrape configs for exporters we do not own — and it means **the set of shipped metrics is the set your monitoring stack already collects**. If you have a ServiceMonitor for an application, that application's metrics ship too. To narrow it, replace `selectAllByDefault: true` in `manifests/vmagent.yaml` with label selectors naming the scrapes you want. Trainy needs the infrastructure ones: kube-state-metrics, node-exporter, kubelet, and the GPU exporter. Worth naming either way: `kube-state-metrics` emits a series per Kubernetes object cluster-wide, so workload **names** (namespace, pod, job) reach Trainy. Not your data — the shape of it. Only namespaces you list are collected. A namespace you create tomorrow is excluded automatically, because it is not on the list — not because a filter caught it. This is enforced in two independent places: the collector's file globs (a log file that is not matched is **never opened**, so those bytes never enter the process) and a namespace filter that fails **closed** on any record whose namespace cannot be resolved. The installer generates both from the single `LOG_NAMESPACES` setting, so they cannot drift apart. The default list is infrastructure only — your GPU operator, networking, monitoring, scheduler and Trainy namespaces. Your workload namespaces are not on it and should not be. Kubernetes events carry pod, job and workload names, so the same default deny applies, enforced on the receiver itself: unlisted namespaces are never watched. Node events are collected separately and deliberately — they are how "the GPU node went away" becomes "the kubelet stopped posting status at 04:12". That watch is constrained by the API server to events whose subject is a **Node**, so events about your pods are never sent to the collector at all, even though the watch is cluster-wide. The `dmesg` DaemonSet runs `dmesg -w` on every node and writes the kernel ring buffer to stdout, where the log shipper forwards it like any other pod log. This is where hardware failure shows up first and in the most detail: Xid faults, PCIe AER corrections, mlx5 link events, NVMe timeouts, soft lockups. None of it appears in any application log. **The container runs privileged**, because reading the kernel ring buffer requires it. Expect this to be the item your security review stops on, so: it runs one command, `dmesg -w`, and nothing else — no host filesystem mounts, no network listeners, no writes. It runs in its own namespace precisely so a pod-security exception for it can be scoped to that one namespace rather than granted where every other Trainy component runs. Kernel logs are node-level, not workload-level. They can name a process that triggered an OOM kill, which is the one place workload detail can appear. Set `INSTALL_DMESG=false` if that is not acceptable — everything else keeps working, and node health detection is unaffected, since those checks read the kernel directly rather than through this DaemonSet. No workload payloads, no application logs, no secrets, no container filesystem contents, no traces, and no exec or shell access. The collectors only push outward — nothing in this install opens a listening port for inbound traffic, and nothing gives Trainy a path into your cluster. ## Prerequisites * Kubernetes cluster with GPU nodes, `kubectl` configured, cluster-admin * `helm` 3.8 or newer * Outbound HTTPS from the cluster to the Trainy ingest endpoint * Your cluster identifier from Trainy * A monitoring stack already running `kube-state-metrics`, `node-exporter` and a GPU metrics exporter, with ServiceMonitors for them `kube-state-metrics` is the one that matters most. Every node health condition reaches Trainy as a `kube_node_status_condition` series from it — without it, the checks run and nobody ever sees the results. `node-exporter` should include the `--collector.infiniband` flag, or RDMA and InfiniBand link metrics will be missing. ## Install ```bash theme={null} git clone https://github.com/Trainy-ai/trainy-telemetry cd trainy-telemetry ``` The bundle is a single directory: the installer, one config file, and the Helm values and manifests it applies. Everything in it is readable, and the values files are commented with what each setting does and why. Nothing is fetched at install time except the upstream Helm charts and container images, all from public registries. ```bash theme={null} cp trainy-telemetry.conf.example trainy-telemetry.conf $EDITOR trainy-telemetry.conf ``` The defaults match Trainy's reference GPU cluster, so on a cluster of the same shape the only setting you must fill in is `CUSTOMER`, the identifier Trainy gave you. The ones most worth a look: | Setting | Why you would change it | | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `KUBE_CONTEXT` | You have more than one cluster configured. | | `GPU_NODE_SELECTOR` | Your GPU nodes carry a different label (AMD clusters usually do). | | `EXPECTED_GPUS` | Your nodes do not have 8 GPUs. | | `RDMA_INTERFACES` / `IB_EXPECTED_ACTIVE_PORTS` | **Expect to set these.** See [Fabric settings](#fabric-settings) — the default inspects every RDMA device, which alarms on the non-fabric ones. | | `LOG_NAMESPACES` / `EVENT_NAMESPACES` | Your infrastructure lives in differently-named namespaces. | | `NODE_HEALTH_MODE` | You want auto-remediation, not just detection — see [below](#node-health-and-auto-remediation). | Keep this file. Re-running the installer with it is how you upgrade or change the install later. ```bash theme={null} ./install.sh --dry-run ``` This changes nothing. It checks your cluster, reports which exporters and ServiceMonitors it found, tests that the ingest endpoints are reachable, and prints everything it would apply — including the generated allowlists and the fully rendered VMAgent. ```bash theme={null} ./install.sh ``` Safe to re-run at any time. It finishes by verifying each component rolled out and listing the node health conditions being published. Tell Trainy the install is up. We will confirm your data is arriving and that ingest is allowlisted for your egress IP. ## Verify ```bash theme={null} ./install.sh --verify # rollout status of every component, no changes ``` Locally: ```bash theme={null} kubectl -n vm-operator get pods kubectl -n vm-operator logs deploy/vmagent-vmagent- --tail=20 kubectl -n otel-collector get pods kubectl -n dmesg-logging get pods # node health checks reporting in: kubectl get nodes -o json | grep -o '"type": *"trainy\.ai/[^"]*"' | sort -u ``` A healthy install shows the operator and VMAgent Deployments, both collectors, and one node-health pod per GPU node, all `Running`. ## GPU metrics Per-GPU health — utilisation, temperature, throttling, XID faults — comes from your GPU metrics exporter, configured through your GPU operator. The installer tells you at preflight whether it found one; if not, enable it: ```bash theme={null} helm upgrade nvidia/gpu-operator \ -n gpu-operator --reuse-values \ --set dcgmExporter.enabled=true ``` If you supply a custom DCGM metrics list, it **replaces** the exporter's whole metric set rather than adding to it. Copying someone else's list will delete whatever they did not have and break your own dashboards. Capture your current metric set first, then add to it. Two things about XID metrics that confuse everyone at least once: * `DCGM_FI_DEV_XID_ERRORS` reads "no data" on a healthy cluster. That is correct, not a misconfiguration: DCGM emits a blank rather than a zero for a GPU with no XID since boot, and the exporter drops blanks, so the series does not exist until a fault occurs. `DCGM_EXP_XID_ERRORS_COUNT` always emits a value — alert on that one. * You are not blind to XID faults without any of this. The node health checks read the kernel ring buffer directly and set a `trainy.ai/GpuXid` condition. DCGM adds per-GPU attribution on top. ## Fabric settings You do not declare InfiniBand versus RoCE anywhere. The checks read `ibstat` and `/sys/class/infiniband` directly and detect what is there — a node with no RDMA devices is a healthy skip, and the RoCE-specific paths (MTU consistency, hardware counters) activate on their own when RoCE devices are present. What you do need to tell it is **which devices carry training traffic**. A typical 8-GPU node exposes roughly 24 mlx5 devices, of which about 16 are fabric rails; the rest are storage, management, or restricted physical functions whose ports are legitimately not Active. Left unset, the port-state check inspects all of them and alarms on every node at once. Trainy sets this explicitly on every cluster it operates, and you should too: ```bash theme={null} # CA names present on a GPU node ibstat -l # which are Active, and InfiniBand vs Ethernet (RoCE) ibv_devinfo | grep -E 'hca_id|link_layer|state' ``` Put the training-fabric devices in `RDMA_INTERFACES`, and set `IB_EXPECTED_ACTIVE_PORTS` to how many of them should be Active on a healthy node — normally the length of that list. Getting the count wrong raises a condition on every node simultaneously, which is the signature of a misconfiguration rather than a fault. Everything else has a working default: expected NIC and NVLink counts, kernel modules, NUMA topology, RoCE counter selection, link-flap thresholds. They are all reachable through `npd.env` if you need them, and the chart's own `values.yaml` documents each one inline. Ask Trainy before changing them — most are tuned against specific failure modes seen in production. Note that NCCL variables for your **training jobs** — `NCCL_IB_HCA`, `NCCL_IB_GID_INDEX`, `NCCL_SOCKET_IFNAME` — are a separate concern and are not set by this install. If you use a fixed GID index on RoCE, be aware that a link flap can leave the conventional index empty with the real address one slot further along, which breaks jobs pinned to it on that device only. ## Node health and auto-remediation `NODE_HEALTH_MODE` has two settings, and **the default acts on your nodes**. **`remediate` (default).** The checks run on your GPU nodes and publish results as `trainy.ai/*` NodeConditions, and the controller acts on them: a faulted node is cordoned, handed to your hardware provider for repair, validated once it returns, and put back into service — escalating to a human when automated repair fails. This is the chart's own default and what Trainy runs on its own clusters. **`detect`.** Checks only. Conditions are still published, still turn into `kube_node_status_condition` metrics, and are still alertable — nothing is cordoned, drained or repaired. Use it for a phased rollout: watch what the checks *would* have acted on for a week, then switch. Which conditions may act on a node is set per condition in `values/node-health.yaml`. Hard faults that make a node unfit for work (XID, ECC, GPU count, fabric manager, NVLink, RDMA link) can cordon it; diagnostic or fleet-wide signals (DNS, AER, MTU, bit error rate, module temperature) report only. That file is where you change the split. Because `remediate` is the default, a plain `./install.sh` gives the controller authority over your GPU nodes. What that authority is bounded by: * `REMEDIATION_MODE` defaults to **`Provider`**, meaning your hardware provider performs the physical repair and the controller only drives the state machine around them — cordon, wait, validate, restore. It does not reboot nodes itself. `InCluster`, which does, is the deliberate exception and is not the default here even though it is the chart's. * `DRAIN_POLICY` defaults to `auto`, but Provider mode does not evict workloads except for the few conditions where the running workload *is* the fault (a GPU still thermally throttling an hour in). Set `askApproval` to gate even those on a human. * Know that the controller **projects** node state: it will revert a manual `kubectl cordon` or `uncordon` on a node it is managing. To take durable manual control of a node, delete its `NodeRemediation` resource. * The installer patches your GPU operator's `ClusterPolicy` so GPU DaemonSets tolerate the remediation taint. Without that, a node under repair reports zero allocatable GPUs and can never pass the validation that would return it to service. A later `helm upgrade` of your GPU operator wipes that patch — re-run the installer afterwards. To start in detect-only and switch later, both directions are a config edit plus a re-run: ```bash theme={null} sed -i 's/^NODE_HEALTH_MODE=.*/NODE_HEALTH_MODE=detect/' trainy-telemetry.conf ./install.sh # controller removed; checks and alerting keep running ``` ## AMD GPU clusters Set `GPU_VENDOR=amd` in your config. Everything else in this guide applies unchanged — metrics, logs, events and kernel logs are vendor-neutral; only the node health checks differ. ``` GPU_VENDOR=amd ``` That layers `values/node-health-amd.yaml` onto the base profile. **The NVIDIA checks do not run on AMD hardware**, and this is not a cosmetic distinction: Xid, DCGM health, NVLink speed, fabric manager and GPU bus enumeration all read NVIDIA interfaces that do not exist on an AMD node. In their place the AMD check set runs: | Runs on AMD | Emits | | -------------- | ------------------------------------------------------------------------------------------------------------------- | | AMD GPU checks | `amd-GpuCount` `amd-GpuDriver` `amd-GpuEcc` `amd-GpuPcie` `amd-Kfd` `amd-Xgmi` `amd-GpuBadPages` `amd-DevicePlugin` | | Vendor-neutral | PCIe ACS/AER, NVMe, DNS, pod-network DNS, egress, kernel modules, thermals, power brake, RDMA NIC count | The Mellanox-specific pieces are off: the `mlxlink` PHY checks and its exporter sidecar need Mellanox Firmware Tools, and the mlx5 kernel-message and link-flap checks read a driver AMD/Pensando NICs do not use. The RoCE checks (`ibPortState`, `roceMtu`, `rdmaGidLayout`) are off too — they self-skip on a node with no RDMA devices rather than failing, so this is about avoiding noise. If your AMD cluster does run RDMA NICs exposing `/sys/class/infiniband`, re-enable `ibPortState` first and confirm it reports cleanly before adding the others. Post-repair validation switches automatically: the controller requests `amd.com/gpu`, runs a HIP burn instead of the NVIDIA one, and the NCCL rung is disabled. The node label matters more than it looks. Helm **merges** map values, so setting an AMD label without removing the NVIDIA one leaves the DaemonSet requiring *both* — and it then schedules on zero nodes, silently, because a DaemonSet with an unsatisfiable selector is not an error. The AMD profile removes the inherited key explicitly. With `GPU_VENDOR=amd` and `GPU_NODE_SELECTOR` left at its default, the installer switches to `feature.node.kubernetes.io/amd-gpu=true` for you. If your AMD nodes carry a different label, set it — and if you set an `nvidia.com/*` label with `GPU_VENDOR=amd`, the installer stops rather than installing something that can never schedule. ## Day-2 operations ```bash Change what is shipped theme={null} $EDITOR trainy-telemetry.conf # e.g. edit LOG_NAMESPACES ./install.sh ``` ```bash Upgrade theme={null} git pull # picks up new pinned chart versions ./install.sh ``` ```bash Pause shipping theme={null} kubectl -n vm-operator scale deploy/vmagent-vmagent- --replicas=0 kubectl -n otel-collector scale deploy/otel-deployment-central --replicas=0 ``` ```bash Remove everything theme={null} ./install.sh --uninstall ``` `--uninstall` removes only what the installer created. Your monitoring stack, your exporters and any namespace it did not create are left alone. Helm never removes CRDs, so the node-health CRDs stay until you delete them explicitly. ## Installing by hand If you would rather not run a script, every component is a plain Helm release or a single manifest, and every file in `values/` carries its exact install command in the header. In short: ```bash theme={null} helm repo add vm https://victoriametrics.github.io/helm-charts/ helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update # metrics: operator, then the agent helm upgrade --install victoria-metrics-operator vm/victoria-metrics-operator \ -n vm-operator --create-namespace --version 0.67.2 \ -f values/vm-operator.yaml # edit manifests/vmagent.yaml, replacing the __PLACEHOLDER__ values, then: kubectl apply -f manifests/vmagent.yaml # logs helm upgrade --install otel-central open-telemetry/opentelemetry-collector \ -n otel-collector --create-namespace --version 0.131.0 \ -f values/otel-logs.yaml \ --set-string 'config.processors.resource/customer.attributes[0].value=' # events helm upgrade --install otel-deployment-central open-telemetry/opentelemetry-collector \ -n otel-collector --version 0.171.0 \ -f values/otel-events.yaml \ --set-string 'config.processors.resource/customer.attributes[0].value=' # node health — checks AND the remediation controller (Provider mode). # Add --set controller.enabled=false for detection only. helm upgrade --install trainy-remediation \ oci://ghcr.io/trainy-ai/charts/trainy-remediation \ -n trainy-system --create-namespace -f values/node-health.yaml \ --set expectedGpus= \ --set-string 'npd.env[0].name=RDMA_INTERFACES' \ --set-string 'npd.env[0].value=' ``` This installs a working remediation pipeline, not just the checks — the chart enables the controller by default and `values/node-health.yaml` does not override it. A faulted GPU node will be cordoned and handed to your provider. Append `--set controller.enabled=false` if you want detection only. Two more differences from the installer worth knowing if you go this route. It does not create a `NodePool`, so repaired nodes are not promoted out of standby automatically. And `drainPolicy` falls back to the chart's `askApproval` rather than the installer's `auto` — more conservative, and it means a drain waits on a human. See [Fabric settings](#fabric-settings) for the per-cluster values above. Installing the log shipper by hand means maintaining the namespace allowlist in **two** places inside `values/otel-logs.yaml` — the file globs and the filter regex — and keeping them identical. They fail differently when they drift: a namespace in the globs only is collected and then dropped; a namespace in the filter only is never read at all. The installer generates both from one list precisely to remove that trap.