Skip to main content
This guide is for the team that operates the Kubernetes cluster. It installs the components Trainy needs to see your cluster’s health, so we can run dashboards, alerts and GPU fault triage against it. The whole install is a handful of Helm releases and one custom resource. It adds no Prometheus, no Grafana, no exporters and no scheduler, and it does not modify the monitoring stack you already run.
Before you start you need two things from Trainy: a cluster identifier (everything you ship is filed under it) and confirmation that your cluster’s egress IP is allowlisted on the ingest endpoint. Without the allowlist the install comes up perfectly healthy and ships nothing.

​
What gets installed

Exporters are not installed. kube-state-metrics, node-exporter and the GPU metrics exporter belong to whoever runs your monitoring stack. This install reads them and reports at preflight which ones it can find.

​
What leaves your cluster

Worth reading before you install, and worth showing to whoever asks.
The agent discovers targets by converting the ServiceMonitors already in your cluster, plus one static job for the GPU exporter. That is what makes the install work without us writing scrape configs for exporters we do not own — and it means the set of shipped metrics is the set your monitoring stack already collects. If you have a ServiceMonitor for an application, that application’s metrics ship too.To narrow it, replace selectAllByDefault: true in manifests/vmagent.yaml with label selectors naming the scrapes you want. Trainy needs the infrastructure ones: kube-state-metrics, node-exporter, kubelet, and the GPU exporter.Worth naming either way: kube-state-metrics emits a series per Kubernetes object cluster-wide, so workload names (namespace, pod, job) reach Trainy. Not your data — the shape of it.
Only namespaces you list are collected. A namespace you create tomorrow is excluded automatically, because it is not on the list — not because a filter caught it.This is enforced in two independent places: the collector’s file globs (a log file that is not matched is never opened, so those bytes never enter the process) and a namespace filter that fails closed on any record whose namespace cannot be resolved. The installer generates both from the single LOG_NAMESPACES setting, so they cannot drift apart.The default list is infrastructure only — your GPU operator, networking, monitoring, scheduler and Trainy namespaces. Your workload namespaces are not on it and should not be.
Kubernetes events carry pod, job and workload names, so the same default deny applies, enforced on the receiver itself: unlisted namespaces are never watched.Node events are collected separately and deliberately — they are how “the GPU node went away” becomes “the kubelet stopped posting status at 04:12”. That watch is constrained by the API server to events whose subject is a Node, so events about your pods are never sent to the collector at all, even though the watch is cluster-wide.
The dmesg DaemonSet runs dmesg -w on every node and writes the kernel ring buffer to stdout, where the log shipper forwards it like any other pod log. This is where hardware failure shows up first and in the most detail: Xid faults, PCIe AER corrections, mlx5 link events, NVMe timeouts, soft lockups. None of it appears in any application log.The container runs privileged, because reading the kernel ring buffer requires it. Expect this to be the item your security review stops on, so: it runs one command, dmesg -w, and nothing else — no host filesystem mounts, no network listeners, no writes. It runs in its own namespace precisely so a pod-security exception for it can be scoped to that one namespace rather than granted where every other Trainy component runs.Kernel logs are node-level, not workload-level. They can name a process that triggered an OOM kill, which is the one place workload detail can appear. Set INSTALL_DMESG=false if that is not acceptable — everything else keeps working, and node health detection is unaffected, since those checks read the kernel directly rather than through this DaemonSet.
No workload payloads, no application logs, no secrets, no container filesystem contents, no traces, and no exec or shell access. The collectors only push outward — nothing in this install opens a listening port for inbound traffic, and nothing gives Trainy a path into your cluster.

​
Prerequisites

  • Kubernetes cluster with GPU nodes, kubectl configured, cluster-admin
  • helm 3.8 or newer
  • Outbound HTTPS from the cluster to the Trainy ingest endpoint
  • Your cluster identifier from Trainy
  • A monitoring stack already running kube-state-metrics, node-exporter and a GPU metrics exporter, with ServiceMonitors for them
kube-state-metrics is the one that matters most. Every node health condition reaches Trainy as a kube_node_status_condition series from it — without it, the checks run and nobody ever sees the results. node-exporter should include the --collector.infiniband flag, or RDMA and InfiniBand link metrics will be missing.

​
Install

1

Get the installer

The bundle is a single directory: the installer, one config file, and the Helm values and manifests it applies. Everything in it is readable, and the values files are commented with what each setting does and why. Nothing is fetched at install time except the upstream Helm charts and container images, all from public registries.
2

Configure

The defaults match Trainy’s reference GPU cluster, so on a cluster of the same shape the only setting you must fill in is CUSTOMER, the identifier Trainy gave you. The ones most worth a look:Keep this file. Re-running the installer with it is how you upgrade or change the install later.
3

Preview

This changes nothing. It checks your cluster, reports which exporters and ServiceMonitors it found, tests that the ingest endpoints are reachable, and prints everything it would apply — including the generated allowlists and the fully rendered VMAgent.
4

Install

Safe to re-run at any time. It finishes by verifying each component rolled out and listing the node health conditions being published.
5

Confirm with Trainy

Tell Trainy the install is up. We will confirm your data is arriving and that ingest is allowlisted for your egress IP.

​
Verify

Locally:
A healthy install shows the operator and VMAgent Deployments, both collectors, and one node-health pod per GPU node, all Running.

​
GPU metrics

Per-GPU health — utilisation, temperature, throttling, XID faults — comes from your GPU metrics exporter, configured through your GPU operator. The installer tells you at preflight whether it found one; if not, enable it:
If you supply a custom DCGM metrics list, it replaces the exporter’s whole metric set rather than adding to it. Copying someone else’s list will delete whatever they did not have and break your own dashboards. Capture your current metric set first, then add to it.
Two things about XID metrics that confuse everyone at least once:
  • DCGM_FI_DEV_XID_ERRORS reads “no data” on a healthy cluster. That is correct, not a misconfiguration: DCGM emits a blank rather than a zero for a GPU with no XID since boot, and the exporter drops blanks, so the series does not exist until a fault occurs. DCGM_EXP_XID_ERRORS_COUNT always emits a value — alert on that one.
  • You are not blind to XID faults without any of this. The node health checks read the kernel ring buffer directly and set a trainy.ai/GpuXid condition. DCGM adds per-GPU attribution on top.

​
Fabric settings

You do not declare InfiniBand versus RoCE anywhere. The checks read ibstat and /sys/class/infiniband directly and detect what is there — a node with no RDMA devices is a healthy skip, and the RoCE-specific paths (MTU consistency, hardware counters) activate on their own when RoCE devices are present. What you do need to tell it is which devices carry training traffic. A typical 8-GPU node exposes roughly 24 mlx5 devices, of which about 16 are fabric rails; the rest are storage, management, or restricted physical functions whose ports are legitimately not Active. Left unset, the port-state check inspects all of them and alarms on every node at once. Trainy sets this explicitly on every cluster it operates, and you should too:
Put the training-fabric devices in RDMA_INTERFACES, and set IB_EXPECTED_ACTIVE_PORTS to how many of them should be Active on a healthy node — normally the length of that list. Getting the count wrong raises a condition on every node simultaneously, which is the signature of a misconfiguration rather than a fault.
Everything else has a working default: expected NIC and NVLink counts, kernel modules, NUMA topology, RoCE counter selection, link-flap thresholds. They are all reachable through npd.env if you need them, and the chart’s own values.yaml documents each one inline. Ask Trainy before changing them — most are tuned against specific failure modes seen in production.
Note that NCCL variables for your training jobs — NCCL_IB_HCA, NCCL_IB_GID_INDEX, NCCL_SOCKET_IFNAME — are a separate concern and are not set by this install. If you use a fixed GID index on RoCE, be aware that a link flap can leave the conventional index empty with the real address one slot further along, which breaks jobs pinned to it on that device only.

​
Node health and auto-remediation

NODE_HEALTH_MODE has two settings, and the default acts on your nodes. remediate (default). The checks run on your GPU nodes and publish results as trainy.ai/* NodeConditions, and the controller acts on them: a faulted node is cordoned, handed to your hardware provider for repair, validated once it returns, and put back into service — escalating to a human when automated repair fails. This is the chart’s own default and what Trainy runs on its own clusters. detect. Checks only. Conditions are still published, still turn into kube_node_status_condition metrics, and are still alertable — nothing is cordoned, drained or repaired. Use it for a phased rollout: watch what the checks would have acted on for a week, then switch. Which conditions may act on a node is set per condition in values/node-health.yaml. Hard faults that make a node unfit for work (XID, ECC, GPU count, fabric manager, NVLink, RDMA link) can cordon it; diagnostic or fleet-wide signals (DNS, AER, MTU, bit error rate, module temperature) report only. That file is where you change the split.
Because remediate is the default, a plain ./install.sh gives the controller authority over your GPU nodes. What that authority is bounded by:
  • REMEDIATION_MODE defaults to Provider, meaning your hardware provider performs the physical repair and the controller only drives the state machine around them — cordon, wait, validate, restore. It does not reboot nodes itself. InCluster, which does, is the deliberate exception and is not the default here even though it is the chart’s.
  • DRAIN_POLICY defaults to auto, but Provider mode does not evict workloads except for the few conditions where the running workload is the fault (a GPU still thermally throttling an hour in). Set askApproval to gate even those on a human.
  • Know that the controller projects node state: it will revert a manual kubectl cordon or uncordon on a node it is managing. To take durable manual control of a node, delete its NodeRemediation resource.
  • The installer patches your GPU operator’s ClusterPolicy so GPU DaemonSets tolerate the remediation taint. Without that, a node under repair reports zero allocatable GPUs and can never pass the validation that would return it to service. A later helm upgrade of your GPU operator wipes that patch — re-run the installer afterwards.
To start in detect-only and switch later, both directions are a config edit plus a re-run:

​
AMD GPU clusters

Set GPU_VENDOR=amd in your config. Everything else in this guide applies unchanged — metrics, logs, events and kernel logs are vendor-neutral; only the node health checks differ.
That layers values/node-health-amd.yaml onto the base profile. The NVIDIA checks do not run on AMD hardware, and this is not a cosmetic distinction: Xid, DCGM health, NVLink speed, fabric manager and GPU bus enumeration all read NVIDIA interfaces that do not exist on an AMD node. In their place the AMD check set runs: The Mellanox-specific pieces are off: the mlxlink PHY checks and its exporter sidecar need Mellanox Firmware Tools, and the mlx5 kernel-message and link-flap checks read a driver AMD/Pensando NICs do not use. The RoCE checks (ibPortState, roceMtu, rdmaGidLayout) are off too — they self-skip on a node with no RDMA devices rather than failing, so this is about avoiding noise. If your AMD cluster does run RDMA NICs exposing /sys/class/infiniband, re-enable ibPortState first and confirm it reports cleanly before adding the others. Post-repair validation switches automatically: the controller requests amd.com/gpu, runs a HIP burn instead of the NVIDIA one, and the NCCL rung is disabled.
The node label matters more than it looks. Helm merges map values, so setting an AMD label without removing the NVIDIA one leaves the DaemonSet requiring both — and it then schedules on zero nodes, silently, because a DaemonSet with an unsatisfiable selector is not an error. The AMD profile removes the inherited key explicitly.With GPU_VENDOR=amd and GPU_NODE_SELECTOR left at its default, the installer switches to feature.node.kubernetes.io/amd-gpu=true for you. If your AMD nodes carry a different label, set it — and if you set an nvidia.com/* label with GPU_VENDOR=amd, the installer stops rather than installing something that can never schedule.

​
Day-2 operations

--uninstall removes only what the installer created. Your monitoring stack, your exporters and any namespace it did not create are left alone. Helm never removes CRDs, so the node-health CRDs stay until you delete them explicitly.

​
Installing by hand

If you would rather not run a script, every component is a plain Helm release or a single manifest, and every file in values/ carries its exact install command in the header. In short:
This installs a working remediation pipeline, not just the checks — the chart enables the controller by default and values/node-health.yaml does not override it. A faulted GPU node will be cordoned and handed to your provider. Append --set controller.enabled=false if you want detection only.
Two more differences from the installer worth knowing if you go this route. It does not create a NodePool, so repaired nodes are not promoted out of standby automatically. And drainPolicy falls back to the chart’s askApproval rather than the installer’s auto — more conservative, and it means a drain waits on a human. See Fabric settings for the per-cluster values above.
Installing the log shipper by hand means maintaining the namespace allowlist in two places inside values/otel-logs.yaml — the file globs and the filter regex — and keeping them identical. They fail differently when they drift: a namespace in the globs only is collected and then dropped; a namespace in the filter only is never read at all. The installer generates both from one list precisely to remove that trap.