Spark

Spark is a distributed computing engine for large-scale data processing. Collection covers Master cluster scheduling status, Worker resource capacity, Driver scheduling health, and Executor memory, GC, and Shuffle I/O.

Configuration¶

Prerequisites¶

• A Spark cluster is deployed;
• DataKit can access the /metrics endpoints exposed by each Spark JVM;
• A JMX Exporter Java Agent compatible with the JVM version is downloaded;
• Exporter ports are exposed only on trusted networks.

Deploy JMX Exporter¶

Use separate Exporter endpoints and rule files for Masters, Workers, Drivers, and Executors. Rules must match the target Spark MBeans and preserve resource ownership tags. When multiple Workers or Executors run on one host, use worker_id and executor_id to distinguish logical instances instead of relying only on host.

Add the Java Agent to the Master and Worker startup options. The following uses the Master and Worker-1 ports from the example mapping:

# Master
export SPARK_MASTER_OPTS="$SPARK_MASTER_OPTS -javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=28099:/opt/jmx/spark-master.yml"

# Worker
export SPARK_WORKER_OPTS="$SPARK_WORKER_OPTS -javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=28199:/opt/jmx/spark-worker.yml"

When submitting an application, add the Java Agent for the Driver and Executor:

spark-submit \
  --conf 'spark.driver.extraJavaOptions=-javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=24099:/opt/jmx/spark-driver.yml' \
  --conf 'spark.executor.extraJavaOptions=-javaagent:/opt/jmx/jmx_prometheus_javaagent.jar=28100:/opt/jmx/spark-executor.yml' \
  ...

After restarting the relevant Spark processes, confirm the endpoints are reachable from the DataKit host:

curl -fsS http://127.0.0.1:28099/metrics | head
curl -fsS http://127.0.0.1:28199/metrics | head

Configure the DataKit Prometheus Collector¶

Open conf.d/prom under the DataKit installation directory and copy prom.conf.sample to spark.conf. One configuration file can contain multiple [[inputs.prom]] collection blocks; centrally manage Master, Worker, Driver, and Executor endpoints.

The following table is a localhost port-mapping example for a three-Worker test cluster. In production, replace it with actual service or service-discovery addresses; the number of Executor endpoints changes with application state.

Example spark.conf:

# Master
[[inputs.prom]]
  urls = ["http://127.0.0.1:28099/metrics"]
  source = "spark-master"
  measurement_name = "spark"
  interval = "10s"
  metric_types = []
  [inputs.prom.tags]
    env = "test"
    component = "master"
    collector = "jmx-exporter"

# Worker-1; copy this block for Worker-2 and Worker-3, and change url and worker_id
[[inputs.prom]]
  urls = ["http://127.0.0.1:28199/metrics"]
  source = "spark-worker"
  measurement_name = "spark"
  interval = "10s"
  metric_types = []
  [inputs.prom.tags]
    env = "test"
    component = "worker"
    collector = "jmx-exporter"
    worker_id = "worker-1"

# Driver
[[inputs.prom]]
  urls = ["http://127.0.0.1:24099/metrics"]
  source = "spark-driver"
  measurement_name = "spark"
  interval = "10s"
  metric_types = []
  [inputs.prom.tags]
    env = "test"
    component = "driver"
    collector = "jmx-exporter"

# Executor-1; copy this block for Executor-2 and Executor-3, and change url, worker_id, and executor_id
[[inputs.prom]]
  urls = ["http://127.0.0.1:28100/metrics"]
  source = "spark-executor"
  measurement_name = "spark"
  interval = "10s"
  metric_types = []
  [inputs.prom.tags]
    env = "test"
    component = "executor"
    collector = "jmx-exporter"
    worker_id = "worker-1"
    executor_id = "1"

urls points to JMX Exporter metric endpoints; source distinguishes collectors; and measurement_name specifies the spark metric set to which metrics are written. Maintain collection blocks for dynamically created or stopped Executors through service discovery or configuration management, and avoid using one static executor_id for multiple instances.

First check on the DataKit host that the endpoints return the expected spark_* metrics:

curl -fsS http://127.0.0.1:28099/metrics | rg '^spark_master_'
curl -fsS http://127.0.0.1:24099/metrics | rg '^spark_driver_'

After configuration, restart DataKit as described in DataKit service management.

Metrics¶

Spark metrics are written to the spark metric set by default. Actual fields are determined by JMX Exporter rules; the following are commonly used operational fields.

Cluster and Worker Capacity¶

Driver Scheduling Health¶

Executor Resources and I/O¶

*_total indicates a cumulative counter. To show throughput or rate, first ensure the counter is continuously reported, then calculate it with rate(); do not place cumulative volume, rate, memory, and task counts on the same axis.