Splunk Linux Use Case

GPU Fleet Watch

AI rigs: thermals, throttle, memory

The scenario

Six AMD boxes are training something and the run is "20 % slower today." Is a card thermal-throttling at 92 °C? Is the PCIe slot negotiation at x4 instead of x16? Did system RAM swap so hard the GPU starved? Is the package cooking at 180 W while the workload asks for 80? On consumer boards with seven GPUs there's no BMC, no datacenter telemetry - the truth is in sysfs, and AMD exposes all of it.

The app recipe

TA-amdgpu AMD GPU telemetry: per-card state from `/sys/class/drm` + KFD topology
TA-throttle CPU package frequency/thermal throttle counters
TA-memwatch RAM headroom, swap thrash (GPU memory pressure is almost always host RAM)
TA-psi IO stall - the dataloader-starved-GPU signature
TA-rapl Chip power, so heat and perf can be correlated

Problems this kills

Killer searches

Who this is for

AI/ML operators running AMD rigs, homelab inference farms, and anyone who has ever had to tell a researcher "it's not your code."

Next

All Use Cases Browse All Apps