Skip to content

Volcengine VCM Kafka

Use the "TrueWatch Cloud Sync" script package in the script market to synchronize cloud monitoring and cloud asset data to TrueWatch

Configuration

Install Func

It is recommended to activate the TrueWatch Integration - Extensions - DataFlux Func (Automata): All prerequisites are automatically installed, please proceed with the script installation.

If you are deploying Func manually, refer to Manual Deployment of Func

Activate Script

Note: Please prepare the required Volcengine AK in advance (for simplicity, you can directly grant global read-only permissions ReadOnlyAccess)

Activate Script in DataFlux Func (Automata)

  1. Log in to the TrueWatch console
  2. Click the【Integration】menu and select【Cloud Account Management】
  3. Click【Add Cloud Account】, select【Volcengine】, and fill in the required information on the interface. If you have already configured the cloud account information before, ignore this step
  4. Click【Test】, and after a successful test, click【Save】. If the test fails, please check if the relevant configuration information is correct and test again
  5. Click【Cloud Account Management】, and you can see the added cloud account in the list. Click the corresponding cloud account to enter the details page
  6. Click the【Integration】button on the cloud account details page, find Volcengine VCM Kafka under the Not Installed list, and click the【Install】button to install it in the pop-up installation interface.

Activate Script Manually

  1. Log in to the Func console, click on【Script Market】, enter the TrueWatch script market, and search for: integration_volcengine_vcm_kafka

  2. Click on【Install】, and enter the corresponding parameters: Volcengine AK ID, AK Secret, and account name.

  3. Click on【Deploy Startup Script】, the system will automatically create a Startup script set and automatically configure the corresponding startup script.

  4. After activation, you can see the corresponding automatic trigger configuration in「Management / Automatic Trigger Configuration」. Click on【Execute】to execute it immediately without waiting for the scheduled time. After a while, you can view the execution task records and corresponding logs.

Verification

  1. In「Management / Automatic Trigger Configuration」, confirm whether the corresponding task has the corresponding automatic trigger configuration, and you can also check the corresponding task records and logs to see if there are any exceptions.
  2. In TrueWatch, check whether the asset information exists in「Infrastructure / Custom」.
  3. In TrueWatch, check whether there is corresponding monitoring data in「Metrics」.

Metrics

After configuring Volcengine Cloud Monitoring, the default Measurement is as follows, and more Metrics can be collected through configuration Volcengine Cloud Monitoring Metrics Details

Note: Need to install monitoring plugin in Volcengine VCM Kafka console

Instance Metrics

The VCM_Kafka namespace includes the following instance Metrics.

MetricName SubNamespace Description Unit Dimension
broker_online_rate Instance Broker Online Rate - Ratio of healthy broker nodes Percent ResourceID
cap_util Instance Capacity Utilization - Instance disk utilization rate Percent ResourceID
connections_count Instance Connection Count - Total TCP connections between clients and server Count ResourceID
connections_per_ip Instance Connections Per IP - Client IP connection count Count ClientIP,ResourceID
consumption_rate_percent Instance Consumption Bandwidth Percent - Network bandwidth utilization for message consumption Percent ResourceID
consumption_request_mean_time Instance Mean Consumption Request Time - Average time for consumption requests Millisecond ResourceID
consumption_request_p99_time Instance Consumption Request P99 Time - P99 latency for consumption requests Millisecond ResourceID
cpu_util Instance CPU Utilization - Instance CPU usage rate Percent ResourceID
disk_util Broker Disk Utilization - Broker disk usage rate Percent ResourceID,Broker
dsik_skewness Instance Disk Skewness - Data distribution imbalance across disks, calculated as (MaxBrokerCap - MinBrokerCap) / MaxBrokerCap Percent ResourceID
group_consumption_rate Group Group Consumption Rate - Message consumption speed of consumer group Count Group,ResourceID
group_msg_accum Group Group Message Accumulation - Unconsumed messages in consumer group. Large accumulation may cause message loss Count Group,ResourceID
group_rebalance_count Group Group Rebalance Count - Historical rebalance count of consumer group Count Group,ResourceID
instance_available_group Instance Available Consumer Groups - Remaining available consumer groups Count ResourceID
instance_available_partition Instance Available Partitions - Remaining available partitions Count ResourceID
max_disk_util Instance Max Disk Utilization - Maximum disk utilization among broker nodes Percent ResourceID
mem_util Instance Memory Utilization - Instance memory usage rate Percent ResourceID
msg_consumption_rate Instance Message Consumption Rate - Data traffic consumed per second Bytes/Second(SI) ResourceID
msg_production_count Instance Message Production Count - Total messages produced per second Count/Second ResourceID
msg_production_rate Instance Message Production Rate - Data traffic produced per second Bytes/Second(SI) ResourceID
partition_msg_accum Group_Partition Partition Message Accumulation - Message accumulation per partition Count Group,Partition,ResourceID,Topic
produce_request_mean_time Broker Mean Produce Request Time - Average time for produce requests Millisecond ResourceID,Broker
production_rate_percent Instance Production Bandwidth Percent - Network bandwidth utilization for message production Percent ResourceID
production_request_mean_time Instance Mean Production Request Time - Average time for production requests Millisecond ResourceID
production_request_p99_time Instance Production Request P99 Time - P99 latency for production requests Millisecond ResourceID
req_consumption_count Instance Request Consumption Count - Consumption request count per second Count/Second ResourceID
req_production_count Instance Request Production Count - Production request count per second Count/Second ResourceID
topic_consumption_rate Group_Topic Topic Consumption Rate - Message consumption speed for topic in consumer group Count Group,ResourceID,Topic
topic_data_size Topic Topic Data Size - Topic data size including all replica data Bits(IEC) ResourceID,Topic
topic_msg_accum Group_Topic Topic Message Accumulation - Unconsumed messages per topic in consumer group Count Group,ResourceID,Topic

Objects

The collected Volcengine VCM Kafka (volcengine_kafka) object data structure can be seen in「Infrastructure - Custom」