Monitoring Configuration

Unique characteristics

If each resource has unique characteristics, use the following syntax for individual configuration:

resources:
  <resource-id>:
    attributes:
      host.name: <hostname>
      host.type: <type>
    <protocol-configuration>

Shared characteristics

If multiple resources share the same characteristics, such as device type, protocols, and credentials, they can be grouped together under a single configuration:

resources:
  <resource-id>:
    attributes:
      host.names: [<hostname1>, <hostname2>, etc.]
      host.type: <type>
    <protocol-configuration>

HTTP

Use the parameters below to configure the HTTP protocol:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: storage
        protocols:
          http:
            https: true
            port: 443
            username: myusername
            password: mypwd
            timeout: 60

ICMP Ping

Use the parameters below to configure the ICMP ping protocol:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: linux
        protocols:
          ping:
            timeout: 10s

IPMI

Use the parameters below to configure the IPMI protocol:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: oob
        protocols:
          ipmi:
            username: myusername
            password: mypwd

OS commands

Use the parameters below to configure OS Commands that are executed locally:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: linux
        protocols:
          osCommand:
            timeout: 120
            useSudo: true
            useSudoCommands: [ cmd1, cmd2 ]
            sudoCommand: sudo

SSH

Use the parameters below to configure the SSH protocol:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: linux
        protocols:
          ssh:
            timeout: 120
            port: 22
            useSudo: true
            useSudoCommands: [ cmd1, cmd2 ]
            sudoCommand: sudo
            username: myusername
            password: mypwd
            privateKey: /tmp/ssh-key.txt

SNMP

Use the parameters below to configure the SNMP protocol:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: linux
        protocols:
          snmp:
            version: v1
            community: public
            port: 161
            timeout: 120s

      myHost2:
        attributes:
          host.name: my-host-02
          host.type: linux
        protocols:
          snmp:
            version: v2c
            community: public
            port: 161
            timeout: 120s

SNMP version 3

Use the parameters below to configure the SNMP version 3 protocol:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost3:
        attributes:
          host.name: my-host-03
          host.type: linux
        protocols:
          snmpv3:
            version: 3
            port: 161
            timeout: 120s
            contextName: myContext
            authType: SHA
            privacy: AES
            username: myUser
            privacyPassword: myPrivacyPassword
            password: myAuthPassword 

WBEM

Use the parameters below to configure the WBEM protocol:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: storage
        protocols:
          wbem:
            protocol: https
            port: 5989
            timeout: 120s
            username: myusername
            password: mypwd

WMI

Use the parameters below to configure the WMI protocol:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: win
        protocols:
          wmi:
            timeout: 120s
            username: myusername
            password: mypwd

WinRM

Use the parameters below to configure the WinRM protocol:

Example

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: win
        protocols:
          winrm:
            protocol: http
            port: 5985
            username: myusername
            password: mypwd
            timeout: 120s
            authentications: [ntlm]

Enterprise Edition authentication

In the Enterprise Edition, the MetricsHub's internal OTLP Exporter authenticates itself with the OpenTelemetry Collector's OTLP gRPC Receiver^[9] by including the HTTP Authorization request header with the credentials.

These settings are already configured in the config/metricshub.yaml file of MetricsHub Enterprise Edition. Changing them is not recommended unless you are familiar with managing communication between the MetricsHub OTLP Exporter and the OpenTelemetry Collector's OTLP Receiver.

To override the default value of the Basic Authentication Header, configure the otel.exporter.otlp.metrics.headers and otel.exporter.otlp.logs.headers parameters under the otel section:

# Internal OpenTelemetry SDK configuration
otel:
  # OpenTelemetry SDK Autoconfigure properties
  # https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure
  # MetricsHub Default configuration
  otel.metrics.exporter: otlp
  otel.exporter.otlp.metrics.endpoint: https://localhost:4317
  otel.exporter.otlp.metrics.protocol: grpc
  otel.exporter.otlp.metrics.headers: Authorization=Basic <base64-username-password>
  otel.exporter.otlp.logs.headers: Authorization=Basic <base64-username-password>
resourceGroups: # ...

where <base64-username-password> credentials are built by first joining your username and password with a colon (myUsername:myPassword) and then encoding the value in base64.

Warning: If you update the Basic Authentication Header, you must generate a new .htpasswd file for the OpenTelemetry Collector Basic Authenticator^[10].

Community Edition authentication

If your OTLP Receiver requires authentication headers, configure the otel.exporter.otlp.metrics.headers and otel.exporter.otlp.logs.headers parameters under the otel section:

otel:
  otel.exporter.otlp.metrics.headers: <custom-header1>
  otel.exporter.otlp.logs.headers: <custom-header2>

resourceGroups: # ...

Collect period

By default, MetricsHub collects metrics from the monitored resources every minute. To change the default collect period:

• For all your resources, add the collectPeriod parameter just before the resourceGroups section:

  collectPeriod: 2m
  
  resourceGroups: # ...
  

• For a specific resource, add the collectPeriod parameter at the resource level. In the example below, we set the collectPeriod to 1m30s for myHost1:

  resourceGroups:
    boston:
      attributes:
        site: boston
      resources:
        myHost1:
          attributes:
            host.name: my-host-01
            host.type: linux
          protocols:
            snmp:
              version: v1
              community: public
              port: 161
              timeout: 120s
          collectPeriod: 1m30s # Customized
  

Warning: Collecting metrics too frequently can cause CPU-intensive workloads.

Connectors

When running MetricsHub, the connectors are automatically selected based on the device type provided and the enabled protocols. However, you have the flexibility to specify which connectors should be utilized or omitted.

The connectors parameter allows you to force, select, or exclude specific connectors. Connector names or category tags should be separated by commas, as illustrated in the example below:

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: win
        protocols:
          wmi:
            timeout: 120s
            username: myusername
            password: mypwd
        connectors: [ "#system" ]

• To force a connector, precede the connector identifier with a plus sign (+), as in +MIB2.
• To exclude a connector from automatic detection, precede the connector identifier with an exclamation mark (!), like !MIB2.
• To stage a connector for processing by automatic detection, configure the connector identifier, for instance, MIB2.
• To stage a category of connectors for processing by automatic detection, precede the category tag with a hash (#), such as #hardware or #system.
• To exclude a category of connectors from automatic detection, precede the category tag to be excluded with an exclamation mark and a hash sign (!#), such as !#system.

Notes:

• Any misspelled connector will be ignored.
• Misspelling a category tag will prevent automatic detection from functioning due to an empty connectors staging.

$ metricshub -l

Patch Connectors

By default, MetricsHub loads connectors from the connectors subdirectory within its installation directory. However, you can extend this functionality by adding a custom directory for additional connectors. This can be done by specifying a patch directory in the metricshub.yaml configuration file.

To configure an additional connector directory, set the patchDirectory property to the path of your custom connectors directory, as shown in the example below:

patchDirectory: /opt/patch/connectors # Replace with the path to your patch connectors directory.

loggerLevel: ...

Configure Connector Variables

In MetricsHub, connector variables are essential for customizing the behavior of data collection. The connector variables are configured in the metricshub.yaml file under the variables section of your configured resource. These variables are specified under the name of the connector to which they belong and contain key-value pairs. The key of each variable corresponds to a variable already configured in the connector.

• Example :

  Below is a configuration using the WindowsProcess connector. The processName variable, defined in the variables section, specifies a list of process names (msedge.exe and metricshub.exe) to monitor:

resources:
  localhost:
    attributes:
      host.name: localhost
      host.type: windows
    protocols:
      wmi:
        timeout: 120
    variables:
      windowsProcess: # Connector ID
        processName: "('msedge.exe', 'metricshub.exe')"

Filter monitors

A monitor is any entity tracked by MetricsHub within the main resource, such as processes, services, storage volumes, or physical devices like disks.

To manage the volume of telemetry data sent to your observability platform and therefore reduce costs and optimize performance, you can specify which monitors to include or exclude.

You can apply monitor inclusion or exclusion in data collection for the following scopes:

• All resources
• All the resources within a specific resource group. A resource group is a container that holds resources to be monitored and generally refers to a site or a specific location.
• A specific resource

This is done by adding the monitorFilters parameter in the relevant section of the config/metricshub.yaml file as described below:

The monitorFilters parameter accepts the following values:

• +<monitor_name> for inclusion
• "!<monitor_name>" for exclusion.

To obtain the monitor name:

1. Refer to the MetricsHub Connector Library^[11]
2. Click the connector of your choice (e.g.: WindowsOS Metrics^[13])
3. Scroll-down to the Metrics section and note down the relevant monitor Type.

Warning: Excluding monitors may lead to missed outage detection or inconsistencies in collected data, such as inaccurate power consumption estimates or other metrics calculated by the engine. Use exclusions carefully to avoid overlooking important information. The monitoring of critical devices such as batteries, power supplies, CPUs, fans, and memories should not be disabled.

monitorFilters: [ +enclosure, +fan, +power_supply ] # Include specific monitors globally
resourceGroups: ...

monitorFilters: [ "!volume" ] # Exclude specific monitors globally

resourceGroups:
  <resource-group-name>:
    monitorFilters: [ +enclosure, +fan, +power_supply ] # Include specific monitors for this group
    resources: ...

resourceGroups:
  <resource-group-name>:
    monitorFilters: [ "!volume" ] # Exclude specific monitors for this group
    resources: ...

resourceGroups:
  <resource-group-name>:
    resources:
      <resource-id>:
        monitorFilters: [ +enclosure, +fan, +power_supply ] # Include specific monitors for this resource

resourceGroups:
  <resource-group-name>:
    resources:
      <resource-id>:
        monitorFilters: [ "!volume" ] # Exclude specific monitors for this resource

Discovery cycle

MetricsHub periodically performs discoveries to detect new components in your monitored environment. By default, MetricsHub runs a discovery after 30 collects. To change this default discovery cycle:

• For all your resources, add the discoveryCycle just before the resourceGroups section:

  discoveryCycle: 15
  
  resourceGroups: # ...
  

• For a specific host, add the discoveryCycle parameter at the resource level and indicate the number of collects after which a discovery will be performed. In the example below, we set the discoveryCycle to be performed after 5 collects for myHost1:

  resourceGroups:
    boston:
      attributes:
        site: boston
      resources:
        myHost1:
          attributes:
            host.name: my-host-01
            host.type: linux
          protocols:
            snmp:
              version: v1
              community: public
              port: 161
              timeout: 120s
          discoveryCycle: 5 # Customized
  

Warning: Running discoveries too frequently can cause CPU-intensive workloads.

Resource Attributes

Add labels in the attributes section to override the data collected by the MetricsHub Agent or add additional attributes to the Host Resource^[14]. These attributes are added to each metric of that Resource when exported to time series platforms like Prometheus.

In the example below, we added a new app attribute and indicated that this is the Jenkins app:

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      myHost1:
        attributes:
          host.name: my-host-01
          host.type: windows
          app: Jenkins
        protocols:
          http:
            https: true
            port: 443
            username: myusername
            password: mypwd
            timeout: 60

Hostname resolution

By default, MetricsHub uses the configured host.name value as-is to populate the Host Resource^[14] attributes. This ensures that the host.name remains consistent with what is configured.

To resolve the host.name to its Fully Qualified Domain Name (FQDN), set the resolveHostnameToFqdn configuration property to true as shown below:

resolveHostnameToFqdn: true

resourceGroups:

This ensures that each configured resource will resolve its host.name to FQDN.

To enable FQDN resolution for a specific resource group, set the resolveHostnameToFqdn property to true under the desired resource group configuration as shown below:

resourceGroups:
  boston:
    resolveHostnameToFqdn: true
    attributes:
      site: boston
    resources:
      # ...

This ensures that all resources within the boston resource group will resolve their host.name to FQDN.

To enable FQDN resolution for an individual resource within a resource group, set the resolveHostnameToFqdn under the resource configuration as shown below:

resourceGroups:
  boston:
    attributes:
      site: boston
    resources:
      my-host-01:
        resolveHostnameToFqdn: true
        attributes:
          host.name: my-host-01
          host.type: linux

In this case, only my-host-01 will resolve its host.name to FQDN, while other resources in the boston group will retain their original host.name values.

Warning: If there is an issue during the resolution, it may result in a different host.name value, potentially impacting metric identity.

Job pool size

By default, MetricsHub runs up to 20 discovery and collect jobs in parallel. To increase or decrease the number of jobs MetricsHub can run simultaneously, add the jobPoolSize parameter just before the resourceGroups section:

jobPoolSize: 40 # Customized

resourceGroups: # ...

Warning: Running too many jobs in parallel can lead to an OutOfMemory error.

Sequential mode

By default, MetricsHub sends the queries to the resource in parallel. Although the parallel mode is faster than the sequential one, too many requests at the same time can lead to the failure of the targeted system.

To force all the network calls to be executed in sequential order:

• For all your resources, add the sequential parameter before the resourceGroups section (NOT RECOMMENDED) and set it to true:

  sequential: true
  
  resourceGroups: # ...
  

• For a specific resource, add the sequential parameter at the resource level and set it to true. In the example below, we enabled the sequential mode for myHost1

  resourceGroups:
    boston:
      attributes:
        site: boston
      resources:
        myHost1:
          attributes:
            host.name: my-host-01
            host.type: linux
          protocols:
            snmp:
              version: v1
              community: public
              port: 161
              timeout: 120s
          sequential: true # Customized
  

Warning: Sending requests in sequential mode slows down the monitoring significantly. Instead of using the sequential mode, you can increase the maximum number of allowed concurrent requests in the monitored system, if the manufacturer allows it.

StateSet metrics compression

By default, MetricsHub compresses StateSet metrics to reduce unnecessary reporting of zero values and to avoid high cardinality in time series databases. This compression can be configured at various levels: globally, per resource group, or for a specific resource.

hw.status{state="ok"} 0
hw.status{state="degraded"} 1
hw.status{state="failed"} 0

hw.status{state="degraded"} 1

Timeout, duration and period format

Timeouts, durations and periods are specified with the below format: