Skip to content

TencentCloud CKafka (Prometheus)

Tencent Cloud CKafka collects Broker, node, and client metrics through Prometheus Exporter, covering message throughput, request latency, connection status, Topic reads/writes, resource levels, and rebalancing to assess CKafka performance and operational stability.

Configuration

Prerequisites

Deploy DataKit in a network that can access the Tencent Cloud CKafka Professional Edition instance. Use a CVM or self-managed host connected to the VPC and subnet selected when obtaining the Prometheus targets. If the host uses security groups or access-control policies, allow the Exporter IP addresses and ports returned by the CKafka console.

Tencent Cloud CKafka Professional Edition provides integration through the open-standard Prometheus Exporter. The console usually returns two types of monitoring targets:

  • Broker JMX Exporter: collects Kafka Broker JMX metrics such as request rate, request latency, Topic read/write rate, Controller, and Log metrics.
  • Node Exporter: collects basic metrics from Broker nodes, including CPU, memory, disk, and network metrics.

Producer, Consumer, Kafka Streams, and Kafka Connect metrics are not exposed by the native CKafka Broker Exporter. To collect client-side metrics, attach JMX Exporter to the corresponding application process and add its /metrics address to the DataKit Prometheus collection configuration.

Obtain Prometheus Monitoring Targets

  1. Sign in to the CKafka console and open the target instance details.
  2. On the instance details page, locate Use Prometheus Monitoring, click Obtain Monitoring Target, select the VPC and subnet to connect, and submit.
  3. After the targets are created, record the Broker JMX Exporter and Node Exporter target addresses returned by the console.

Example monitoring targets:

Type Example Address Description
Broker JMX Exporter <jmx-exporter-ip>:60001, <jmx-exporter-ip>:60003 Collects Broker JMX metrics.
Node Exporter <node-exporter-ip>:60002, <node-exporter-ip>:60004 Collects basic Broker node metrics.

If the CKafka instance is migrated, resized, or moved to another Availability Zone, the underlying Brokers may change. Obtain the latest Exporter IP addresses and ports from the console again.

Configure DataKit

  • In the DataKit installation directory, go to conf.d/samples, copy prom.conf.sample, and name it ckafka.conf.

cp prom.conf.sample ckafka.conf

  • Edit ckafka.conf.
[[inputs.prom]]
  urls = [
    "http://<jmx-exporter-ip>:60001/metrics",
    "http://<jmx-exporter-ip>:60003/metrics",
    "http://<jmx-exporter-ip>:60005/metrics",
    "http://<jmx-exporter-ip>:60007/metrics",
  ]
  source = "kafka_jmx"
  metric_name_filter = ["^kafka_"]
  measurement_prefix = ""
  keep_exist_metric_name = false
  honor_timestamps = true
  election = false
  disable_host_tag = true
  disable_instance_tag = true
  disable_info_tag = false

  [[inputs.prom.measurements]]
    prefix = "kafka_controller_"
    name = "kafka_controller"

  [[inputs.prom.measurements]]
    prefix = "kafka_network_"
    name = "kafka_network"

  [[inputs.prom.measurements]]
    prefix = "kafka_log_"
    name = "kafka_log"

  [[inputs.prom.measurements]]
    prefix = "kafka_server_"
    name = "kafka_server"

  [inputs.prom.tags]
    cloud_provider = "tencentcloud"
    service = "ckafka"
    region_id = "<region-id>"
    cluster_id = "<ckafka-instance-id>"
    cluster_name = "<ckafka-instance-name>"


[[inputs.prom]]
  urls = [
    "http://<node-exporter-ip>:60002/metrics",
    "http://<node-exporter-ip>:60004/metrics",
    "http://<node-exporter-ip>:60006/metrics",
    "http://<node-exporter-ip>:60008/metrics",
  ]
  source = "kafka_node"
  metric_name_filter = ["^node_"]
  measurement_prefix = ""
  keep_exist_metric_name = false
  honor_timestamps = true
  election = false
  disable_host_tag = true
  disable_instance_tag = true
  disable_info_tag = false

  [[inputs.prom.measurements]]
    prefix = "node_"
    name = "kafka_node"

  [inputs.prom.tags]
    cloud_provider = "tencentcloud"
    service = "ckafka"
    region_id = "<region-id>"
    cluster_id = "<ckafka-instance-id>"
    cluster_name = "<ckafka-instance-name>"


# Optional: collect client-side JMX Exporter metrics from applications such as Producer, Consumer, Kafka Streams, and Kafka Connect.
[[inputs.prom]]
  urls = [
    "http://<producer-client-ip>:7072/metrics",
    "http://<consumer-client-ip>:7073/metrics",
    "http://<streams-client-ip>:7074/metrics",
    "http://<connect-worker-ip>:7075/metrics",
  ]
  source = "kafka_client"
  metric_name_filter = ["^kafka_(producer|consumer|stream|connect)_"]
  measurement_prefix = ""
  keep_exist_metric_name = false
  honor_timestamps = true
  election = false
  disable_host_tag = false
  disable_instance_tag = true
  disable_info_tag = false

  [[inputs.prom.measurements]]
    prefix = "kafka_producer_"
    name = "kafka_producer"

  [[inputs.prom.measurements]]
    prefix = "kafka_consumer_"
    name = "kafka_consumer"

  [[inputs.prom.measurements]]
    prefix = "kafka_connect_"
    name = "kafka_connect"

  [[inputs.prom.measurements]]
    prefix = "kafka_stream_"
    name = "kafka_stream"

  [inputs.prom.tags]
    cloud_provider = "tencentcloud"
    service = "ckafka"
    region_id = "<region-id>"
    cluster_id = "<ckafka-instance-id>"
    cluster_name = "<ckafka-instance-name>"
  • Restart DataKit

Run the following command:

datakit service -R

Metrics

After configuring Prometheus collection, the recommended scope includes 9 measurements and 111 metric contexts distinguished by measurement, field, and fixed semantic tags. The monitoring targets returned by the Tencent Cloud CKafka console primarily provide Broker JMX and Node Exporter metrics; Producer, Consumer, Kafka Streams, and Kafka Connect metrics require JMX Exporter on the corresponding client processes.

Controller Metrics

kafka_controller contains 4 recommended metric contexts.

MetricName MetricDescribe Dimensions Unit
KafkaController_Value{name=ActiveControllerCount} Number of active Controllers in the cluster; normally 1. cloud_provider, cluster_name, host count
KafkaController_Value{name=GlobalPartitionCount} Current total partition count; use the maximum when multiple Brokers expose the same global value. host count
KafkaController_Value{name=GlobalTopicCount} Current total Topic count; use the maximum when multiple Brokers expose the same global value. host count
KafkaController_Value{name=OfflinePartitionsCount} Number of partitions without a Leader that cannot be read or written normally. cloud_provider, cluster_name, host count

Network Request Metrics

kafka_network contains 23 recommended metric contexts.

MetricName MetricDescribe Dimensions Unit
RequestChannel_Value{name=RequestQueueSize} Entries currently waiting in the Broker request queue. host count
RequestChannel_Value{name=ResponseQueueSize} Entries currently waiting in the Broker response queue. host count
RequestMetrics_95thPercentile{name=TotalTimeMs;request=FetchConsumer} FetchConsumer request end-to-end P95 processing time. cloud_provider, cluster_name, host ms
RequestMetrics_95thPercentile{name=TotalTimeMs;request=Produce} Produce request end-to-end P95 processing time. cloud_provider, cluster_name, host ms
RequestMetrics_Mean{name=LocalTimeMs;request=FetchConsumer} FetchConsumer request average local processing time. host ms
RequestMetrics_Mean{name=LocalTimeMs;request=FetchFollower} FetchFollower request average local processing time. host ms
RequestMetrics_Mean{name=MessageConversionsTimeMs;request=Produce} Produce request average message-format conversion time. host ms
RequestMetrics_Mean{name=RemoteTimeMs;request=FetchConsumer} FetchConsumer request average remote processing time. host ms
RequestMetrics_Mean{name=RemoteTimeMs;request=FetchFollower} FetchFollower request average remote processing time. host ms
RequestMetrics_Mean{name=RemoteTimeMs;request=Produce} Produce request average remote processing time. host ms
RequestMetrics_Mean{name=RequestQueueTimeMs;request=FetchConsumer} FetchConsumer request average wait time in the request queue. host ms
RequestMetrics_Mean{name=RequestQueueTimeMs;request=Produce} Produce request average wait time in the request queue. host ms
RequestMetrics_Mean{name=ResponseQueueTimeMs;request=FetchConsumer} FetchConsumer request average wait time in the response queue. host ms
RequestMetrics_Mean{name=ResponseQueueTimeMs;request=FetchFollower} FetchFollower request average wait time in the response queue. host ms
RequestMetrics_Mean{name=ResponseQueueTimeMs;request=Produce} Produce request average wait time in the response queue. host ms
RequestMetrics_Mean{name=ResponseSendTimeMs;request=FetchConsumer} FetchConsumer request average response-send time. host ms
RequestMetrics_Mean{name=ResponseSendTimeMs;request=FetchFollower} FetchFollower request average response-send time. host ms
RequestMetrics_Mean{name=ResponseSendTimeMs;request=Produce} Produce request average response-send time. host ms
RequestMetrics_Mean{name=TotalTimeMs;request=FetchConsumer} FetchConsumer request average end-to-end processing time. host ms
RequestMetrics_Mean{name=TotalTimeMs;request=Produce} Produce request average end-to-end processing time. host ms
RequestMetrics_OneMinuteRate{name=RequestsPerSec} Broker request rate over the last minute. host, request ops
SocketServer_Value{name=MemoryPoolUsed} Memory currently used by the Broker network-layer memory pool. host B
SocketServer_Value{name=NetworkProcessorAvgIdlePercent} Ratio of time that Broker network-processing threads are idle. host percent

Log Metrics

kafka_log contains 4 recommended metric contexts.

MetricName MetricDescribe Dimensions Unit
LogFlushStats_OneMinuteRate{name=LogFlushRateAndTimeMs} Average number of Broker log flush operations per second. host ops
Log_Value{name=LogEndOffset} Current log end offset of the selected Topic partition. host, partition, topic offset
Log_Value{name=LogStartOffset} Earliest readable log offset of the selected Topic partition. host, partition, topic offset
Log_Value{name=Size} Disk space used by logs for the selected Topic partition. host, topic B

Broker Service Metrics

kafka_server contains 9 recommended metric contexts.

MetricName MetricDescribe Dimensions Unit
BrokerTopicMetrics_OneMinuteRate{name=BytesInPerSec} Bytes of produced messages received by the Broker or Topic per second. host, topic B/S
BrokerTopicMetrics_OneMinuteRate{name=BytesOutPerSec} Bytes sent by the Broker or Topic to consumers per second. host, topic B/S
BrokerTopicMetrics_OneMinuteRate{name=FetchMessageConversionsPerSec} Number of Fetch message-format conversions performed by the Broker per second. host, topic ops
BrokerTopicMetrics_OneMinuteRate{name=MessagesInPerSec} Messages received by the Broker or Topic per second. host, topic ops
BrokerTopicMetrics_OneMinuteRate{name=TotalFetchRequestsPerSec} Fetch requests received by the Broker or Topic per second. host, topic ops
BrokerTopicMetrics_OneMinuteRate{name=TotalProduceRequestsPerSec} Produce requests received by the Broker or Topic per second. host, topic ops
KafkaRequestHandlerPool_FiveMinuteRate{name=RequestHandlerAvgIdlePercent} Average idle ratio of Broker request-handler threads over the last five minutes. cloud_provider, cluster_name, host percent
ReplicaManager_Value{name=UnderMinIsrPartitionCount} Number of partitions whose in-sync replica count is below min.insync.replicas. cloud_provider, cluster_name count
ReplicaManager_Value{name=UnderReplicatedPartitions} Number of partitions whose replica count is below the target replication factor. cloud_provider, cluster_name count

Node Resource Metrics

kafka_node contains 11 recommended metric contexts.

MetricName MetricDescribe Dimensions Unit
cpu_seconds_total{mode=idle} Broker node CPU utilization derived from the idle CPU-time increase. host percent
disk_read_bytes_total Bytes read from Broker node disks per second. host B/S
disk_reads_completed_total Disk read operations completed by Broker nodes per second. host ops
disk_writes_completed_total Disk write operations completed by Broker nodes per second. host ops
disk_written_bytes_total Bytes written to Broker node disks per second. host B/S
load1 Broker node system load average over the last 1 minute. host -
load15 Broker node system load average over the last 15 minutes. host -
load5 Broker node system load average over the last 5 minutes. host -
memory_MemTotal_bytes Broker node memory utilization calculated from total and available memory. - percent
network_receive_bytes_total{device!=lo} Bytes received per second by non-loopback NICs on Broker nodes. host B/S
network_transmit_bytes_total{device!=lo} Bytes transmitted per second by non-loopback NICs on Broker nodes. host B/S

Producer Client Metrics

kafka_producer contains 13 recommended metric contexts.

MetricName MetricDescribe Dimensions Unit
metrics_buffer_available_bytes Currently unused Producer buffer memory. client_id, host B
metrics_buffer_exhausted Cumulative record sends discarded by Producer due to buffer exhaustion. client_id, host count
metrics_buffer_exhausted_rate Producer record sends discarded per second due to buffer exhaustion. client_id, host ops
metrics_bufferpool_wait_time_ns Cumulative time Producer waited for buffer-space allocation. client_id, host ns
metrics_connection_count Current active client connections. client_id, host count
metrics_failed_authentication Cumulative client connections with authentication failures. client_id, host count
metrics_flush_time_ns Cumulative time spent by Producer on flush operations. client_id, host ns
metrics_io_wait_time_ns Cumulative time Producer I/O threads waited for readable/writable sockets. client_id, host ns
metrics_request_rate Requests sent by the client per second. client_id, host ops
metrics_requests_in_flight Producer requests sent but not yet answered. client_id, host count
metrics_response_rate Responses received by the client per second. client_id, host ops
metrics_select_rate Select operations performed by client I/O threads per second. client_id, host ops
metrics_txn_commit_time_ns Cumulative time Producer spent committing transactions. client_id, host ns

Consumer Client Metrics

kafka_consumer contains 14 recommended metric contexts.

MetricName MetricDescribe Dimensions Unit
coordinator_metrics_rebalance_latency Total duration of the latest Consumer Group rebalance. client_id, host ms
coordinator_metrics_rebalance_rate_per_hour Average Consumer Group rebalances per hour. client_id, host times/hour
metrics_connection_close_rate Connections closed by Consumer per second. client_id, host ops
metrics_connection_count Current active client connections. client_id, host count
metrics_failed_authentication Cumulative client connections with authentication failures. client_id, host count
metrics_incoming_byte_rate Bytes read by the client from network sockets per second. client_id, host B/S
metrics_io_time_ns_avg Average I/O time per Consumer select call. client_id, host ns
metrics_io_wait_time_ns_avg Average time Consumer I/O threads wait for readable/writable sockets. client_id, host ns
metrics_outgoing_byte_rate Bytes sent by the client to Brokers per second. client_id, host B/S
metrics_request_rate Requests sent by the client per second. client_id, host ops
metrics_request_size_avg Average Consumer request size. client_id, host B
metrics_response_rate Responses received by the client per second. client_id, host ops
metrics_select_rate Select operations performed by client I/O threads per second. client_id, host ops
metrics_time_between_poll_avg Average interval between adjacent Consumer poll calls. client_id, host ms

Kafka Streams Metrics

kafka_stream contains 25 recommended metric contexts.

MetricName MetricDescribe Dimensions Unit
stream_state_metrics_all_rate Operations performed by Kafka Streams state stores per second. host, rocksdb_state_id, thread_id ops
stream_state_metrics_block_cache_capacity Kafka Streams RocksDB block-cache capacity. host, rocksdb_state_id, thread_id B
stream_state_metrics_block_cache_data_hit_ratio Kafka Streams RocksDB block-cache data hit ratio. host, rocksdb_state_id, thread_id percent
stream_state_metrics_block_cache_filter_hit_ratio Kafka Streams RocksDB block-cache filter hit ratio. host, rocksdb_state_id, thread_id percent
stream_state_metrics_block_cache_index_hit_ratio Kafka Streams RocksDB block-cache index hit ratio. host, rocksdb_state_id, thread_id percent
stream_state_metrics_block_cache_usage Current Kafka Streams RocksDB block-cache usage. host, rocksdb_state_id, thread_id B
stream_state_metrics_bytes_read_compaction_rate Bytes read per second during Kafka Streams state-store compaction. host, rocksdb_state_id, thread_id B/S
stream_state_metrics_bytes_read_rate Bytes read by Kafka Streams state stores per second. host, rocksdb_state_id, thread_id B/S
stream_state_metrics_bytes_written_compaction_rate Bytes written per second during Kafka Streams state-store compaction. host, rocksdb_state_id, thread_id B/S
stream_state_metrics_bytes_written_rate Bytes written by Kafka Streams state stores per second. host, rocksdb_state_id, thread_id B/S
stream_task_metrics_active_process_ratio Ratio of Kafka Streams task-thread time spent processing records. host, task_id, thread_id percent
stream_task_metrics_cache_size_bytes Total size of the Kafka Streams task cache. host, task_id, thread_id B
stream_task_metrics_enforced_processing_rate Enforced processing operations performed by Kafka Streams tasks per second. host, task_id, thread_id ops
stream_task_metrics_process_rate Records processed by Kafka Streams tasks per second. host, task_id, thread_id ops
stream_thread_metrics_blocked_time_ns Cumulative time Kafka Streams threads were blocked while waiting. host, thread_id ns
stream_thread_metrics_commit_rate State commits by Kafka Streams threads per second. host, thread_id ops
stream_thread_metrics_poll_latency_avg Average latency of Kafka Streams thread poll calls. host, thread_id ms
stream_thread_metrics_poll_rate Poll operations performed by Kafka Streams threads per second. host, thread_id ops
stream_thread_metrics_task Cumulative tasks created by Kafka Streams threads. host, thread_id count
stream_thread_metrics_task_closed Cumulative tasks closed by Kafka Streams threads. host, thread_id count
stream_thread_metrics_thread_start_time Kafka Streams thread start timestamp. host, thread_id ms
stream_topic_metrics_bytes_consumed Cumulative bytes consumed by Kafka Streams from Topics. host, topic B
stream_topic_metrics_bytes_produced Cumulative bytes produced by Kafka Streams to Topics. host, topic B
stream_topic_metrics_records_consumed Cumulative records consumed by Kafka Streams source processor nodes. host, processor_node_id, topic count
stream_topic_metrics_records_produced Cumulative records produced by Kafka Streams sink processor nodes. host, processor_node_id, topic count

Kafka Connect Metrics

kafka_connect contains 8 recommended metric contexts.

MetricName MetricDescribe Dimensions Unit
worker_connector_count Number of Connectors currently running in the Kafka Connect Worker. host count
worker_connector_startup_attempts Cumulative Connector startup attempts by the Kafka Connect Worker. host count
worker_connector_startup_failure_percentage Kafka Connect Worker Connector startup failure ratio. host percent
worker_rebalance_completed_rebalances Cumulative rebalances completed by the Kafka Connect Worker. host count
worker_rebalance_time_since_last_rebalance_ms Time since the Kafka Connect Worker last completed a rebalance. host ms
worker_task_startup_attempts Cumulative Task startup attempts by the Kafka Connect Worker. host count
worker_task_startup_failure Cumulative Task startup failures in the Kafka Connect Worker. host count
worker_task_startup_failure_percentage Kafka Connect Worker Task startup failure ratio. host percent