Huawei Cloud ROMA
Collect Huawei Cloud ROMA Metrics data
Configuration¶
Install Func¶
It is recommended to activate TrueWatch Integration - Extensions - DataFlux Func (Automata): all prerequisites are automatically installed, please proceed with the script installation
If deploying Func manually, refer to Manual Func Deployment
Install Script¶
Note: Please prepare the Huawei Cloud AK that meets the requirements in advance (for simplicity, you can directly grant the global read-only permission
ReadOnlyAccess
)
-
Log in to the Func console, click 【Script Market】, enter the TrueWatch Script Market, and search for
ID: integration_huaweicloud_roma
-
Click 【Install】, then enter the corresponding parameters: Huawei Cloud AK, SK, and account name
-
Click 【Deploy Startup Script】, the system will automatically create the
Startup
script set and configure the corresponding startup scripts -
After enabling, you can see the corresponding automatic trigger configuration in 「Manage / Automatic Trigger Configuration」. Click 【Execute】 to immediately execute it once without waiting for the scheduled time. Wait a moment, and you can view the execution task records and corresponding logs
Verification¶
- In 「Manage / Automatic Trigger Configuration」, confirm whether the corresponding task has the automatic trigger configuration, and check the corresponding task records and logs for any exceptions
- In TrueWatch, 「Infrastructure - Resource Catalog」, check if the asset information exists
- In TrueWatch, 「Metrics」, check if there is corresponding monitoring data
Metrics¶
Collect Huawei Cloud ROMA Metrics data, more metrics can be collected through configuration Huawei Cloud ROMA Metrics Details
Instance Monitoring Metrics¶
Metric Name | Metric Description | Unit | Dimensions |
---|---|---|---|
current_partitions | This metric is used to count the number of partitions already used in the Kafka instance | Count | instance_id |
current_topics | This metric is used to count the number of topics already created in the Kafka instance | Count | instance_id |
group_msgs | This metric is used to count the total number of backlogged messages in all consumer groups in the Kafka instance | Count | instance_id |
Node Monitoring Metrics¶
Metric Name | Metric Description | Unit | Dimensions |
---|---|---|---|
broker_data_size | This metric is used to count the current message data size of the node | Byte | instance_id |
broker_messages_in_rate | This metric is used to count the number of messages produced per second | Count/s | instance_id |
broker_bytes_in_rate | This metric is used to count the number of bytes produced per second | Byte/s | instance_id |
broker_bytes_out_rate | This metric is used to count the number of bytes consumed per second | Byte/s | instance_id |
broker_public_bytes_in_rate | Counts the public network access inflow traffic per second of the Broker node | Byte/s | instance_id |
broker_public_bytes_out_rate | Counts the public network access outflow traffic per second of the Broker node | Byte/s | instance_id |
broker_fetch_mean | Counts the average duration of consumer request processing by the Broker node | ms | instance_id |
broker_produce_mean | Average duration of production request processing | ms | instance_id |
broker_cpu_core_load | Average load of each CPU core collected at the virtual machine level of the Kafka node | % | instance_id |
broker_disk_usage | Disk capacity usage rate collected at the virtual machine level of the Kafka node | % | instance_id |
broker_memory_usage | Memory usage rate collected at the virtual machine level of the Kafka node | % | instance_id |
broker_heap_usage | Heap memory usage rate collected from the Kafka process JVM of the Kafka node | % | instance_id |
broker_alive | Indicates whether the Kafka node is alive | 1: Alive 0: Offline | instance_id |
broker_connections | Current number of all TCP connections on the Kafka node | Count | instance_id |
broker_cpu_usage | CPU usage rate of the Kafka node virtual machine | % | instance_id |
broker_total_bytes_in_rate | Network access inflow traffic per second of the Broker node | Byte/s | instance_id |
broker_total_bytes_out_rate | Network access outflow traffic per second of the Broker node | Byte/s | instance_id |
broker_disk_read_rate | Disk read operation traffic | Byte/s | instance_id |
broker_disk_write_rate | Disk write operation traffic | Byte/s | instance_id |
network_bandwidth_usage | Network bandwidth utilization rate | % | instance_id |
Consumer Group Monitoring Metrics¶
Metric Name | Metric Description | Unit | Dimensions |
---|---|---|---|
messages_consumed | This metric is used to count the number of messages already consumed by the current consumer group | Count | instance_id |
messages_remained | This metric is used to count the number of messages that can be consumed by the consumer group | Count | instance_id |
topic_messages_remained | This metric is used to count the number of messages that can be consumed by the consumer group for the specified queue | Count | instance_id |
topic_messages_consumed | This metric is used to count the number of messages already consumed by the consumer group for the specified queue | Count | instance_id |
consumer_messages_remained | This metric is used to count the number of messages that can be consumed by the consumer group | Count | instance_id |
consumer_messages_consumed | This metric is used to count the number of messages already consumed by the consumer group | Count | instance_id |
Objects¶
The collected Huawei Cloud ROMA object data structure can be seen in 「Infrastructure - Resource Catalog」
{
"measurement": "huaweicloud_SYS.ROMA",
"tags": {
"RegionId" : "cn-north-4",
"charging_mode" : "1",
"connect_address" : "192.168.0.161,192.168.0.126,192.168.0.31",
"description" : "",
"engine" : "kafka",
"engine_version" : "2.7",
"instance_id" : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
"kafka_manager_user" : "",
"name" : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
"port" : "9092",
"project_id" : "f5f4c067d68xxxx86e173b18367bf",
"resource_spec_code" : "",
"service_type" : "advanced",
"specification" : "kafka.2u4g.cluster.small * 3 broker",
"status" : "RUNNING",
"storage_type" : "hec",
"user_id" : "e4b27d49128e4bd0893b28d032a2e7c0",
"user_name" : "xxxx"
},
"fields": {
"created_at" : "1693203968959",
"maintain_begin" : "02:00:00",
"maintain_end" : "06:00:00",
"storage_space" : 186,
"total_storage_space" : 300,
"message" : "{Instance JSON Data}"
}
}
Note: Fields in
tags
,fields
may change with subsequent updatesNote 1: The value of
tags.name
is the instance ID, used as the unique identifierNote 2: The following fields are all JSON serialized strings
```