Huawei Cloud ROMA
Collect Huawei Cloud ROMA Metrics data
Configuration¶
Install Func¶
It is recommended to activate TrueWatch Integration - Extensions - DataFlux Func (Automata): all prerequisites are automatically installed, please proceed with the script installation
If deploying Func manually, refer to Manual Func Deployment
Install Script¶
Note: Please prepare the Huawei Cloud AK that meets the requirements in advance (for simplicity, you can directly grant the global read-only permission
ReadOnlyAccess)
-
Log in to the Func console, click 【Script Market】, enter the TrueWatch Script Market, and search for
ID: integration_huaweicloud_roma -
Click 【Install】, then enter the corresponding parameters: Huawei Cloud AK, SK, and account name
-
Click 【Deploy Startup Script】, the system will automatically create the
Startupscript set and configure the corresponding startup scripts -
After enabling, you can see the corresponding automatic trigger configuration in 「Manage / Automatic Trigger Configuration」. Click 【Execute】 to immediately execute it once without waiting for the scheduled time. Wait a moment, and you can view the execution task records and corresponding logs
Verification¶
- In 「Manage / Automatic Trigger Configuration」, confirm whether the corresponding task has the automatic trigger configuration, and check the corresponding task records and logs for any exceptions
- In TrueWatch, 「Infrastructure - Resource Catalog」, check if the asset information exists
- In TrueWatch, 「Metrics」, check if there is corresponding monitoring data
Metrics¶
Collect Huawei Cloud ROMA Metrics data, more metrics can be collected through configuration Huawei Cloud ROMA Metrics Details
Instance Monitoring Metrics¶
| Metric Name | Metric Description | Unit | Dimensions |
|---|---|---|---|
| current_partitions | This metric is used to count the number of partitions already used in the Kafka instance | Count | instance_id |
| current_topics | This metric is used to count the number of topics already created in the Kafka instance | Count | instance_id |
| group_msgs | This metric is used to count the total number of backlogged messages in all consumer groups in the Kafka instance | Count | instance_id |
Node Monitoring Metrics¶
| Metric Name | Metric Description | Unit | Dimensions |
|---|---|---|---|
| broker_data_size | This metric is used to count the current message data size of the node | Byte | instance_id |
| broker_messages_in_rate | This metric is used to count the number of messages produced per second | Count/s | instance_id |
| broker_bytes_in_rate | This metric is used to count the number of bytes produced per second | Byte/s | instance_id |
| broker_bytes_out_rate | This metric is used to count the number of bytes consumed per second | Byte/s | instance_id |
| broker_public_bytes_in_rate | Counts the public network access inflow traffic per second of the Broker node | Byte/s | instance_id |
| broker_public_bytes_out_rate | Counts the public network access outflow traffic per second of the Broker node | Byte/s | instance_id |
| broker_fetch_mean | Counts the average duration of consumer request processing by the Broker node | ms | instance_id |
| broker_produce_mean | Average duration of production request processing | ms | instance_id |
| broker_cpu_core_load | Average load of each CPU core collected at the virtual machine level of the Kafka node | % | instance_id |
| broker_disk_usage | Disk capacity usage rate collected at the virtual machine level of the Kafka node | % | instance_id |
| broker_memory_usage | Memory usage rate collected at the virtual machine level of the Kafka node | % | instance_id |
| broker_heap_usage | Heap memory usage rate collected from the Kafka process JVM of the Kafka node | % | instance_id |
| broker_alive | Indicates whether the Kafka node is alive | 1: Alive 0: Offline | instance_id |
| broker_connections | Current number of all TCP connections on the Kafka node | Count | instance_id |
| broker_cpu_usage | CPU usage rate of the Kafka node virtual machine | % | instance_id |
| broker_total_bytes_in_rate | Network access inflow traffic per second of the Broker node | Byte/s | instance_id |
| broker_total_bytes_out_rate | Network access outflow traffic per second of the Broker node | Byte/s | instance_id |
| broker_disk_read_rate | Disk read operation traffic | Byte/s | instance_id |
| broker_disk_write_rate | Disk write operation traffic | Byte/s | instance_id |
| network_bandwidth_usage | Network bandwidth utilization rate | % | instance_id |
Consumer Group Monitoring Metrics¶
| Metric Name | Metric Description | Unit | Dimensions |
|---|---|---|---|
| messages_consumed | This metric is used to count the number of messages already consumed by the current consumer group | Count | instance_id |
| messages_remained | This metric is used to count the number of messages that can be consumed by the consumer group | Count | instance_id |
| topic_messages_remained | This metric is used to count the number of messages that can be consumed by the consumer group for the specified queue | Count | instance_id |
| topic_messages_consumed | This metric is used to count the number of messages already consumed by the consumer group for the specified queue | Count | instance_id |
| consumer_messages_remained | This metric is used to count the number of messages that can be consumed by the consumer group | Count | instance_id |
| consumer_messages_consumed | This metric is used to count the number of messages already consumed by the consumer group | Count | instance_id |
Objects¶
The collected Huawei Cloud ROMA object data structure can be seen in 「Infrastructure - Resource Catalog」
{
"measurement": "huaweicloud_SYS.ROMA",
"tags": {
"RegionId" : "cn-north-4",
"charging_mode" : "1",
"connect_address" : "192.168.0.161,192.168.0.126,192.168.0.31",
"description" : "",
"engine" : "kafka",
"engine_version" : "2.7",
"instance_id" : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
"kafka_manager_user" : "",
"name" : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
"port" : "9092",
"project_id" : "f5f4c067d68xxxx86e173b18367bf",
"resource_spec_code" : "",
"service_type" : "advanced",
"specification" : "kafka.2u4g.cluster.small * 3 broker",
"status" : "RUNNING",
"storage_type" : "hec",
"user_id" : "e4b27d49128e4bd0893b28d032a2e7c0",
"user_name" : "xxxx"
},
"fields": {
"created_at" : "1693203968959",
"maintain_begin" : "02:00:00",
"maintain_end" : "06:00:00",
"storage_space" : 186,
"total_storage_space" : 300,
"message" : "{Instance JSON Data}"
}
}
Note: Fields in
tags,fieldsmay change with subsequent updatesNote 1: The value of
tags.nameis the instance ID, used as the unique identifierNote 2: The following fields are all JSON serialized strings
```