AWS MSK (Prometheus)¶

Amazon Managed Streaming for Apache Kafka (Amazon MSK) open monitoring exposes Kafka Broker JMX and Broker node metrics in Prometheus format. DataKit scrapes these endpoints directly with the Prometheus collector, without AWS AK, Func, or CloudWatch collection scripts.

Configuration¶

Prerequisites¶

• This integration applies to MSK Provisioned clusters with Amazon MSK open monitoring enabled.
• An existing cluster must be in the ACTIVE state before open monitoring can be enabled. Enable it during cluster creation or later in the Monitoring section of the cluster properties.
• DataKit must be deployed in a network that can resolve and access the private DNS names of MSK Brokers, typically the same VPC or a connected VPC.
• MSK security groups must allow the DataKit security group or network segment to access TCP ports 11001 and 11002; DNS must be enabled for the VPC. For network requirements, see AWS MSK Prometheus Collector prerequisites.
• JMX Exporter uses port 11001, and Node Exporter uses port 11002. AWS recommends a scrape interval of at least 60 seconds; shorter intervals may increase cluster CPU utilization.

Note: KRaft metadata mode and MSK Express Broker cannot enable open monitoring and public access at the same time. Open monitoring itself is free, but cross-Availability Zone data transfer may incur charges.

Enable Open Monitoring¶

When creating an MSK Provisioned cluster, select Enable open monitoring with Prometheus in the Monitoring section and enable JMX Exporter, Node Exporter, or both.

Enable it for an existing cluster as follows:

1. Sign in to the Amazon MSK console and open the target cluster.
2. On the Properties tab, locate Monitoring and click Edit.
3. Select Enable open monitoring with Prometheus.
4. Enable JMX Exporter, Node Exporter, or both, and save the changes.

You can also use the AWS CLI to update the monitoring configuration:

aws kafka update-monitoring \
  --cluster-arn <cluster-arn> \
  --current-version <current-version> \
  --open-monitoring 'Prometheus={JmxExporter={EnabledInBroker=true},NodeExporter={EnabledInBroker=true}}'

For more information, see the official AWS documentation:

• Enable open monitoring on a new cluster
• Enable open monitoring on an existing cluster

Obtain Prometheus Monitoring Targets¶

Use ListNodes to obtain the cluster Broker DNS names:

aws kafka list-nodes \
  --cluster-arn <cluster-arn>

Record all Broker DNS names in BrokerNodeInfo.Endpoints from the response. Configure two targets for each Broker:

To collect Controller JMX metrics from a KRaft cluster, also add the Controller DNS names returned by ControllerNodeInfo.Endpoints to the 11001 targets. For endpoint formats and KRaft details, see AWS Prometheus host configuration.

Verify network connectivity from the DataKit host:

curl -sS "http://<broker-dns>:11001/metrics" | head
curl -sS "http://<broker-dns>:11002/metrics" | head

Configure DataKit¶

In the DataKit installation directory, go to conf.d/samples, copy prom.conf.sample, and name it aws_msk_prom.conf:

cp prom.conf.sample aws_msk_prom.conf

Edit aws_msk_prom.conf. Add all Broker DNS names to the corresponding urls; add KRaft Controller DNS names only to the JMX Exporter configuration.

[[inputs.prom]]
  urls = [
    "http://<broker-dns-1>:11001/metrics",
    "http://<broker-dns-2>:11001/metrics",
    # Optional for KRaft:
    # "http://<controller-dns-1>:11001/metrics",
  ]
  source = "kafka_jmx"
  interval = "60s"
  metric_name_filter = ["^kafka_"]
  measurement_prefix = ""
  keep_exist_metric_name = false
  honor_timestamps = true
  election = false
  disable_host_tag = true
  disable_instance_tag = true
  disable_info_tag = false

  [[inputs.prom.measurements]]
    prefix = "kafka_controller_"
    name = "kafka_controller"

  [[inputs.prom.measurements]]
    prefix = "kafka_network_"
    name = "kafka_network"

  [[inputs.prom.measurements]]
    prefix = "kafka_log_"
    name = "kafka_log"

  [[inputs.prom.measurements]]
    prefix = "kafka_server_"
    name = "kafka_server"

  [inputs.prom.tags]
    cloud_provider = "aws"
    service = "msk"
    region_id = "<aws-region>"
    cluster_id = "<cluster-arn-or-id>"
    cluster_name = "<msk-cluster-name>"


[[inputs.prom]]
  urls = [
    "http://<broker-dns-1>:11002/metrics",
    "http://<broker-dns-2>:11002/metrics",
  ]
  source = "kafka_node"
  interval = "60s"
  metric_name_filter = ["^node_"]
  measurement_prefix = ""
  keep_exist_metric_name = false
  honor_timestamps = true
  election = false
  disable_host_tag = true
  disable_instance_tag = true
  disable_info_tag = false

  [[inputs.prom.measurements]]
    prefix = "node_"
    name = "kafka_node"

  [inputs.prom.tags]
    cloud_provider = "aws"
    service = "msk"
    region_id = "<aws-region>"
    cluster_id = "<cluster-arn-or-id>"
    cluster_name = "<msk-cluster-name>"


# Optional: collect JMX Exporter metrics from application clients such as Producer, Consumer, Kafka Streams, and Kafka Connect.
[[inputs.prom]]
  urls = [
    "http://<producer-client-ip>:7072/metrics",
    "http://<consumer-client-ip>:7073/metrics",
    "http://<streams-client-ip>:7074/metrics",
    "http://<connect-worker-ip>:7075/metrics",
  ]
  source = "kafka_client"
  interval = "60s"
  metric_name_filter = ["^kafka_(producer|consumer|stream|connect)_"]
  measurement_prefix = ""
  keep_exist_metric_name = false
  honor_timestamps = true
  election = false
  disable_host_tag = false
  disable_instance_tag = true
  disable_info_tag = false

  [[inputs.prom.measurements]]
    prefix = "kafka_producer_"
    name = "kafka_producer"

  [[inputs.prom.measurements]]
    prefix = "kafka_consumer_"
    name = "kafka_consumer"

  [[inputs.prom.measurements]]
    prefix = "kafka_connect_"
    name = "kafka_connect"

  [[inputs.prom.measurements]]
    prefix = "kafka_stream_"
    name = "kafka_stream"

  [inputs.prom.tags]
    cloud_provider = "aws"
    service = "msk"
    region_id = "<aws-region>"
    cluster_id = "<cluster-arn-or-id>"
    cluster_name = "<msk-cluster-name>"

11001 exposes Kafka Broker JMX and Consumer Lag metrics, while 11002 exposes Broker node metrics such as CPU and disk. For the exact scope, see AWS Prometheus metrics. Producer, Consumer, Kafka Streams, and Kafka Connect are client-process metrics and require JMX Exporter on the corresponding processes.

Restart DataKit after completing the configuration:

datakit service -R

Verification¶

1. Confirm in the DataKit logs that aws_msk_prom.conf has no connection or resolution errors.
2. In TrueWatch Metrics, confirm that measurements such as kafka_controller, kafka_network, kafka_server, and kafka_node exist.
3. Verify that cloud_provider=aws, cluster_name=<msk-cluster-name>, and the Broker host tag have the expected values.
4. If client JMX Exporters are configured, confirm that kafka_producer, kafka_consumer, kafka_stream, and kafka_connect contain data.

Metrics¶

After configuring Prometheus collection, the recommended scope includes 9 measurements and 115 metric contexts distinguished by measurement, field, and fixed semantic tags. Amazon MSK open monitoring natively provides Broker JMX and Node Exporter metrics; Producer, Consumer, Kafka Streams, and Kafka Connect metrics require JMX Exporter on the corresponding client processes.

Controller Metrics¶

kafka_controller contains 8 recommended metric contexts.

Network Request Metrics¶

kafka_network contains 24 recommended metric contexts.

Log Metrics¶

kafka_log contains 4 recommended metric contexts.

Broker Service Metrics¶

kafka_server contains 10 recommended metric contexts.

Node Resource Metrics¶

kafka_node contains 9 recommended metric contexts.

Producer Client Metrics¶

kafka_producer contains 13 recommended metric contexts.

Consumer Client Metrics¶

kafka_consumer contains 14 recommended metric contexts.

Kafka Streams Metrics¶

kafka_stream contains 25 recommended metric contexts.

Kafka Connect Metrics¶

kafka_connect contains 8 recommended metric contexts.