Volcengine VCM Kafka¶

Volcengine VCM Kafka metrics include broker online rate, CPU utilization, memory utilization, disk utilization, message production and consumption rates, and request latency. These metrics reflect the operating status, resource usage, and performance of Kafka instances.

Configuration¶

Install Func¶

It is recommended to activate the TrueWatch Integration - Extensions - DataFlux Func (Automata): All prerequisites are automatically installed, please proceed with the script installation.

If you are deploying Func manually, refer to Manual Deployment of Func

Install Script¶

Note: Prepare a Volcengine AK that meets the requirements in advance (for simplicity, you can grant the global read-only permission ReadOnlyAccess).

To synchronize monitoring data for VCM Kafka cloud resources, install the corresponding collection script: "TrueWatch Integration (Volcengine-VCM Kafka Collection)" (ID: integration_volcengine_vcm_kafka).

After clicking 【Install】, enter the corresponding parameters: Volcengine AK and Volcengine account name.

Click 【Deploy Startup Script】. The system automatically creates a Startup script set and configures the corresponding startup script.

After it is enabled, you can find the corresponding automatic trigger configuration under 「Management / Automatic Trigger Configuration」. Click 【Execute】 to run it immediately without waiting for the scheduled time. After a short wait, you can view the execution records and corresponding logs.

To collect the corresponding logs, also enable the relevant log collection script. To collect billing data, enable the cloud billing collection script.

Verification¶

In「Management / Automatic Trigger Configuration」, confirm whether the corresponding task has the corresponding automatic trigger configuration, and you can also check the corresponding task records and logs to see if there are any exceptions.
In TrueWatch, check whether the asset information exists in「Infrastructure / Custom」.
In TrueWatch, check whether there is corresponding monitoring data in「Metrics」.

Metrics¶

After configuring Volcengine Cloud Monitoring, the default Measurement is as follows, and more Metrics can be collected through configuration Volcengine Cloud Monitoring Metrics Details

Note: Need to install monitoring plugin in Volcengine VCM Kafka console

`MetricName`	`Subnamespace`	Metric Name	MetricUnit	Dimension
`MetricName`	`SubNamespace`	Metric Name	MetricUnit	Dimension
`broker_online_rate`	`Instance`	Broker Online Rate - Ratio of healthy broker nodes	Percent	ResourceID
`cap_util`	`Instance`	Capacity Utilization - Instance disk utilization rate	Percent	ResourceID
`connections_count`	`Instance`	Connection Count - Total TCP connections between clients and server	Count	ResourceID
`connections_per_ip`	`Instance`	Connections Per IP - Client IP connection count	Count	ClientIP,ResourceID
`consumption_rate_percent`	`Instance`	Consumption Bandwidth Percent - Network bandwidth utilization for message consumption	Percent	ResourceID
`consumption_request_mean_time`	`Instance`	Mean Consumption Request Time - Average time for consumption requests	Millisecond	ResourceID
`consumption_request_p99_time`	`Instance`	Consumption Request P99 Time - P99 latency for consumption requests	Millisecond	ResourceID
`cpu_util`	`Instance`	CPU Utilization - Instance CPU usage rate	Percent	ResourceID
`disk_util`	`Broker`	Disk Utilization - Broker disk usage rate	Percent	ResourceID,Broker
`dsik_skewness`	`Instance`	Disk Skewness - Data distribution imbalance across disks, calculated as (MaxBrokerCap - MinBrokerCap) / MaxBrokerCap	Percent	ResourceID
`group_consumption_rate`	`Group`	Group Consumption Rate - Message consumption speed of consumer group	Count	Group,ResourceID
`group_msg_accum`	`Group`	Group Message Accumulation - Unconsumed messages in consumer group. Large accumulation may cause message loss	Count	Group,ResourceID
`group_rebalance_count`	`Group`	Group Rebalance Count - Historical rebalance count of consumer group	Count	Group,ResourceID
`instance_available_group`	`Instance`	Available Consumer Groups - Remaining available consumer groups	Count	ResourceID
`instance_available_partition`	`Instance`	Available Partitions - Remaining available partitions	Count	ResourceID
`max_disk_util`	`Instance`	Max Disk Utilization - Maximum disk utilization among broker nodes	Percent	ResourceID
`mem_util`	`Instance`	Memory Utilization - Instance memory usage rate	Percent	ResourceID
`msg_consumption_rate`	`Instance`	Message Consumption Rate - Data traffic consumed per second	Bytes/Second(SI)	ResourceID
`msg_production_count`	`Instance`	Message Production Count - Total messages produced per second	Count/Second	ResourceID
`msg_production_rate`	`Instance`	Message Production Rate - Data traffic produced per second	Bytes/Second(SI)	ResourceID
`partition_msg_accum`	`Group_Partition`	Partition Message Accumulation - Message accumulation per partition	Count	Group,Partition,ResourceID,Topic
`produce_request_mean_time`	`Broker`	Mean Produce Request Time - Average time for produce requests	Millisecond	ResourceID,Broker
`production_rate_percent`	`Instance`	Production Bandwidth Percent - Network bandwidth utilization for message production	Percent	ResourceID
`production_request_mean_time`	`Instance`	Mean Production Request Time - Average time for production requests	Millisecond	ResourceID
`production_request_p99_time`	`Instance`	Production Request P99 Time - P99 latency for production requests	Millisecond	ResourceID
`req_consumption_count`	`Instance`	Request Consumption Count - Consumption request count per second	Count/Second	ResourceID
`req_production_count`	`Instance`	Request Production Count - Production request count per second	Count/Second	ResourceID
`topic_consumption_rate`	`Group_Topic`	Topic Consumption Rate - Message consumption speed for topic in consumer group	Count	Group,ResourceID,Topic
`topic_data_size`	`Topic`	Topic Data Size - Topic data size including all replica data	Bits(IEC)	ResourceID,Topic
`topic_msg_accum`	`Group_Topic`	Topic Message Accumulation - Unconsumed messages per topic in consumer group	Count	Group,ResourceID,Topic

Objects¶

The collected Volcengine VCM Kafka (volcengine_kafka) object data structure can be seen in「Infrastructure - Custom」