Hadoop HDFS NameNode¶

Collect HDFS namenode metrics information.

Installation and Configuration¶

Since NameNode is developed in Java, metrics can be collected using the jmx-exporter approach.

1. NameNode Configuration¶

1.1 Download jmx-exporter¶

Download URL: https://github.com/prometheus/jmx_exporter

1.2 Download jmx script¶

Download URL: https://github.com/lrwh/jmx-exporter/blob/main/hadoop-hdfs-namenode.yml

1.3 Adjust NameNode Startup Parameters¶

Add the following to the namenode startup parameters:

{JAVA_GC_ARGS} -javaagent:/opt/jmx/jmx_exporter-1.0.1.jar=localhost:17107:/opt/jmx/hadoop-hdfs-namenode.yml

1.4 Restart NameNode¶

2. DataKit Collector Configuration¶

2.1 Install DataKit¶

2.2 Configure Collector¶

Since jmx-exporter can directly expose the metrics URL, it can be collected directly using the prom collector.

Navigate to the conf.d/prom directory under the DataKit installation directory, and copy prom.conf.sample to namenode.conf.

cp prom.conf.sample namenode.conf

Adjust the contents of namenode.conf as follows:

  urls = ["http://localhost:17107/metrics"]
  source ="hdfs-namenode"
  [inputs.prom.tags]
    component = "hdfs-namenode" 
  interval = "10s"

Adjust other configurations as needed

, parameter adjustment instructions:

• urls: The jmx-exporter metrics URL, fill in the metrics URL exposed by the corresponding component
• source: Collector alias, recommended to distinguish
• keep_exist_metric_name: Keep metric names
• interval: Collection interval
• inputs.prom.tags: Add additional tags

3. Restart DataKit¶

Restart DataKit

Metrics¶

Hadoop Metrics¶

NameNode metrics are under the Hadoop metrics, here we mainly introduce NameNode related metrics