NVIDIA DCGM Exporter (Prometheus)

Description

This connector scrapes NVIDIA dcgm-exporter Prometheus metrics over HTTP(S).

gpu hardware nvidia

Enterprise Connector

This connector requires the Enterprise edition of MetricsHub.

Target

Typical platform: Nvidia

Operating systems: Linux, Out-Of-Band

Prerequisites

Leverages: NVIDIA DCGM Exporter

Technology and protocols: HTTP

Examples

CLI

metricshub HOSTNAME -t linux -c +NvidiaDCGMExporter --https --http-port 443 -u USERNAME

metricshub.yaml

resourceGroups:
  <RESOURCE_GROUP>:
    resources:
      <HOSTNAME-ID>:
        attributes:
          host.name: <HOSTNAME> # Change with actual host name
          host.type: linux
        connectors: [ +NvidiaDCGMExporter ] # Optional, to load only this connector
        protocols:
          http:
            https: true
            port: 443 # or probably something else
            username: <USERNAME> # Change with actual credentials
            password: <PASSWORD> # Encrypted using metricshub-encrypt

Connector Activation Criteria

The NVIDIA DCGM Exporter (Prometheus) connector will be automatically activated, and its status will be reported as OK if all the below criteria are met:

• The HTTP Request below to the managed host succeeds:
  • get /metrics
  • The response body contains: DCGM_FI_DEV_GPU_UTIL (regex)

Metrics

Type	Collected Metrics	Specific Attributes
gpu	hw.energy{hw.type="gpu"} hw.errors{hw.type="gpu", hw.error.type="ecc-memory-corrected", hw.error.severity="non-critical"} hw.errors{hw.type="gpu", hw.error.type="ecc-memory-uncorrected", hw.error.severity="critical"} hw.errors{hw.type="gpu", hw.error.type="pcie-replay"} hw.errors{hw.type="gpu", hw.error.type="xid"} hw.gpu.bar1.limit hw.gpu.bar1.usage{hw.gpu.bar1.state="free"} hw.gpu.bar1.usage{hw.gpu.bar1.state="used"} hw.gpu.clock.limit{hw.type="memory"} hw.gpu.clock.limit{hw.type="sm"} hw.gpu.clock.limit{hw.type="video"} hw.gpu.clock.reduction.time{hw.gpu.clock.reference="application"} hw.gpu.clock.reduction.time{hw.gpu.clock.reference="base"} hw.gpu.clock{hw.type="memory"} hw.gpu.clock{hw.type="sm"} hw.gpu.clock{hw.type="video"} hw.gpu.fabric.condition.active{hw.gpu.fabric.condition.type="access-timeout-recovery"} hw.gpu.fabric.condition.active{hw.gpu.fabric.condition.type="bandwidth-degraded"} hw.gpu.fabric.condition.active{hw.gpu.fabric.condition.type="route-recovery"} hw.gpu.fabric.condition.active{hw.gpu.fabric.condition.type="route-unhealthy"} hw.gpu.fabric.configuration.status{state="gpu-state-invalid|incompatible-gpu-firmware|incorrect-chassis-serial-number|incorrect-system-guid|insufficient-nvlinks|invalid-location|no-partition|ok"} hw.gpu.fabric.partition.assigned hw.gpu.io{hw.gpu.direction="receive", hw.gpu.interface="pcie"} hw.gpu.io{hw.gpu.direction="transmit", hw.gpu.interface="pcie"} hw.gpu.io{hw.gpu.interface="nvlink"} hw.gpu.memory.limit hw.gpu.memory.usage{hw.gpu.memory.state="free"} hw.gpu.memory.usage{hw.gpu.memory.state="reserved"} hw.gpu.memory.usage{hw.gpu.memory.state="used"} hw.gpu.memory.utilization hw.gpu.occupancy hw.gpu.pcie.link.generation hw.gpu.pcie.link.generation.limit{limit_type="max"} hw.gpu.pcie.link.width hw.gpu.pcie.link.width.limit{limit_type="max"} hw.gpu.performance_state hw.gpu.throttle.time{hw.gpu.throttle.reason="board-limit"} hw.gpu.throttle.time{hw.gpu.throttle.reason="hardware-power-brake"} hw.gpu.throttle.time{hw.gpu.throttle.reason="hardware-thermal"} hw.gpu.throttle.time{hw.gpu.throttle.reason="low-utilization"} hw.gpu.throttle.time{hw.gpu.throttle.reason="reliability"} hw.gpu.throttle.time{hw.gpu.throttle.reason="software-power-cap"} hw.gpu.throttle.time{hw.gpu.throttle.reason="software-thermal"} hw.gpu.throttle.time{hw.gpu.throttle.reason="sync-boost"} hw.gpu.utilization{hw.gpu.task="decoder"} hw.gpu.utilization{hw.gpu.task="dram"} hw.gpu.utilization{hw.gpu.task="encoder"} hw.gpu.utilization{hw.gpu.task="fp16"} hw.gpu.utilization{hw.gpu.task="fp32"} hw.gpu.utilization{hw.gpu.task="fp64"} hw.gpu.utilization{hw.gpu.task="general"} hw.gpu.utilization{hw.gpu.task="graphics-engine"} hw.gpu.utilization{hw.gpu.task="integer"} hw.gpu.utilization{hw.gpu.task="memory-copy"} hw.gpu.utilization{hw.gpu.task="sm"} hw.gpu.utilization{hw.gpu.task="tensor"} hw.power.limit{hw.type="gpu"} hw.power{hw.type="gpu"} hw.status{hw.type="gpu", hw.gpu.subsystem="fabric", state="degraded|failed|ok"} hw.status{hw.type="gpu", hw.gpu.subsystem="fabric-manager", state="degraded|failed|ok"} hw.status{hw.type="gpu", state="present"} hw.temperature{sensor_location="gpu"} hw.temperature{sensor_location="gpu-memory"}	host.name hw.gpu.fabric.clique.id hw.gpu.fabric.cluster.uuid hw.gpu.index hw.gpu.model hw.gpu.nvidia.mig_instance_id hw.gpu.pci.bus_id hw.gpu.uuid hw.id hw.name id name type