Hadoop HDFS DataNode¶

Collect HDFS datanode Metrics information.

Installation and Deployment¶

Since the DataNode is developed in the LANGUAGE, it is possible to use jmx-exporter to collect Metrics information.

1. DataNode Configuration¶

1.1 Download jmx-exporter¶

Download address: https://github.com/prometheus/jmx_exporter

1.2 Download jmx Script¶

Download address: https://github.com/lrwh/jmx-exporter/blob/main/hadoop-hdfs-datanode.yml

1.3 Adjust DataNode Startup Parameters¶

Add the following to the startup parameters of the datanode:

{JAVA_GC_ARGS} -javaagent:/opt/guance/jmx/jmx_exporter-1.0.1.jar=localhost:17106:/opt/guance/jmx/hadoop-hdfs-datanode.yml

1.4 Restart DataNode¶

2. DataKit Collector Configuration¶

2.1 Install DataKit¶

2.2 Configure the Collector¶

jmx-exporter can directly expose metrics url, so the prom collector can be used for collection.

Go to the DataKit installation directory under conf.d/prom, copy prom.conf.sample as datanode.conf.

cp prom.conf.sample datanode.conf

Adjust the content of datanode.conf as follows:

  urls = ["http://localhost:17106/metrics"]
  source = "hdfs-datanode"
  [inputs.prom.tags]
    component = "hdfs-datanode" 
  interval = "10s"

Other configurations should be adjusted as needed

, parameter adjustment instructions:

• urls: The jmx-exporter Metrics address, fill in the corresponding component's exposed Metrics url here.
• source: Alias for the collector, recommended to distinguish.
• keep_exist_metric_name: Keep metric names.
• interval: Collection interval.
• inputs.prom.tags: Add extra tags.

3. Restart DataKit¶

Restart DataKit

Metrics¶

Hadoop Measurement¶

DataNode Metrics are located under the Hadoop Measurement set, mainly introducing the explanation of DataNode related Metrics.