Hadoop HDFS NameNode¶

Collect HDFS namenode Metrics information.

Installation and Deployment¶

Since the NameNode is developed in the java LANGUAGE, it is possible to use the jmx-exporter method to collect Metrics information.

1. NameNode Configuration¶

1.1 Download jmx-exporter¶

Download address: https://github.com/prometheus/jmx_exporter

1.2 Download jmx Script¶

Download address: https://github.com/lrwh/jmx-exporter/blob/main/hadoop-hdfs-namenode.yml

1.3 Adjust NameNode Startup Parameters¶

Add the following to the startup parameters of namenode:

{JAVA_GC_ARGS} -javaagent:/opt/guance/jmx/jmx_exporter-1.0.1.jar=localhost:17107:/opt/guance/jmx/hadoop-hdfs-namenode.yml

1.4 Restart NameNode¶

2. DataKit Collector Configuration¶

2.1 Install DataKit¶

2.2 Configure Collector¶

The jmx-exporter can directly expose metrics url, so it can be collected directly through the prom collector.

Go to the conf.d/prom under the DataKit installation directory, copy prom.conf.sample to namenode.conf.

cp prom.conf.sample namenode.conf

Adjust the content of namenode.conf as follows:

  urls = ["http://localhost:17107/metrics"]
  source ="hdfs-namenode"
  [inputs.prom.tags]
    component = "hdfs-namenode"
  interval = "10s"

Other configurations can be adjusted as needed

, adjustment parameter description :

• urls: The jmx-exporter Metrics address, fill in the Metrics url exposed by the corresponding component here
• source: Collector alias, it's recommended to make distinctions
• keep_exist_metric_name: Keep the metric name
• interval: Collection interval
• inputs.prom.tags: Add extra tags

3. Restart DataKit¶

Restart DataKit

Metrics¶

Hadoop Measurement Sets¶

NameNode Metrics are located under the Hadoop Measurement sets, here we mainly introduce the related NameNode Metrics descriptions.