Skip to content

Huawei Cloud ROMA

Collect Huawei Cloud ROMA Metrics data

Configuration

Install Func

It is recommended to activate TrueWatch Integration - Extensions - DataFlux Func (Automata): all prerequisites are automatically installed, please proceed with the script installation

If deploying Func manually, refer to Manual Func Deployment

Install Script

Note: Please prepare the Huawei Cloud AK that meets the requirements in advance (for simplicity, you can directly grant the global read-only permission ReadOnlyAccess)

  1. Log in to the Func console, click 【Script Market】, enter the TrueWatch Script Market, and search for ID: integration_huaweicloud_roma

  2. Click 【Install】, then enter the corresponding parameters: Huawei Cloud AK, SK, and account name

  3. Click 【Deploy Startup Script】, the system will automatically create the Startup script set and configure the corresponding startup scripts

  4. After enabling, you can see the corresponding automatic trigger configuration in 「Manage / Automatic Trigger Configuration」. Click 【Execute】 to immediately execute it once without waiting for the scheduled time. Wait a moment, and you can view the execution task records and corresponding logs

Verification

  1. In 「Manage / Automatic Trigger Configuration」, confirm whether the corresponding task has the automatic trigger configuration, and check the corresponding task records and logs for any exceptions
  2. In TrueWatch, 「Infrastructure - Resource Catalog」, check if the asset information exists
  3. In TrueWatch, 「Metrics」, check if there is corresponding monitoring data

Metrics

Collect Huawei Cloud ROMA Metrics data, more metrics can be collected through configuration Huawei Cloud ROMA Metrics Details

Instance Monitoring Metrics

Metric Name Metric Description Unit Dimensions
current_partitions This metric is used to count the number of partitions already used in the Kafka instance Count instance_id
current_topics This metric is used to count the number of topics already created in the Kafka instance Count instance_id
group_msgs This metric is used to count the total number of backlogged messages in all consumer groups in the Kafka instance Count instance_id

Node Monitoring Metrics

Metric Name Metric Description Unit Dimensions
broker_data_size This metric is used to count the current message data size of the node Byte instance_id
broker_messages_in_rate This metric is used to count the number of messages produced per second Count/s instance_id
broker_bytes_in_rate This metric is used to count the number of bytes produced per second Byte/s instance_id
broker_bytes_out_rate This metric is used to count the number of bytes consumed per second Byte/s instance_id
broker_public_bytes_in_rate Counts the public network access inflow traffic per second of the Broker node Byte/s instance_id
broker_public_bytes_out_rate Counts the public network access outflow traffic per second of the Broker node Byte/s instance_id
broker_fetch_mean Counts the average duration of consumer request processing by the Broker node ms instance_id
broker_produce_mean Average duration of production request processing ms instance_id
broker_cpu_core_load Average load of each CPU core collected at the virtual machine level of the Kafka node % instance_id
broker_disk_usage Disk capacity usage rate collected at the virtual machine level of the Kafka node % instance_id
broker_memory_usage Memory usage rate collected at the virtual machine level of the Kafka node % instance_id
broker_heap_usage Heap memory usage rate collected from the Kafka process JVM of the Kafka node % instance_id
broker_alive Indicates whether the Kafka node is alive 1: Alive 0: Offline instance_id
broker_connections Current number of all TCP connections on the Kafka node Count instance_id
broker_cpu_usage CPU usage rate of the Kafka node virtual machine % instance_id
broker_total_bytes_in_rate Network access inflow traffic per second of the Broker node Byte/s instance_id
broker_total_bytes_out_rate Network access outflow traffic per second of the Broker node Byte/s instance_id
broker_disk_read_rate Disk read operation traffic Byte/s instance_id
broker_disk_write_rate Disk write operation traffic Byte/s instance_id
network_bandwidth_usage Network bandwidth utilization rate % instance_id

Consumer Group Monitoring Metrics

Metric Name Metric Description Unit Dimensions
messages_consumed This metric is used to count the number of messages already consumed by the current consumer group Count instance_id
messages_remained This metric is used to count the number of messages that can be consumed by the consumer group Count instance_id
topic_messages_remained This metric is used to count the number of messages that can be consumed by the consumer group for the specified queue Count instance_id
topic_messages_consumed This metric is used to count the number of messages already consumed by the consumer group for the specified queue Count instance_id
consumer_messages_remained This metric is used to count the number of messages that can be consumed by the consumer group Count instance_id
consumer_messages_consumed This metric is used to count the number of messages already consumed by the consumer group Count instance_id

Objects

The collected Huawei Cloud ROMA object data structure can be seen in 「Infrastructure - Resource Catalog」

{
  "measurement": "huaweicloud_SYS.ROMA",
  "tags": {      
    "RegionId"           : "cn-north-4",
    "charging_mode"      : "1",
    "connect_address"    : "192.168.0.161,192.168.0.126,192.168.0.31",
    "description"        : "",
    "engine"             : "kafka",
    "engine_version"     : "2.7",
    "instance_id"        : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "kafka_manager_user" : "",
    "name"               : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "port"               : "9092",
    "project_id"         : "f5f4c067d68xxxx86e173b18367bf",
    "resource_spec_code" : "",
    "service_type"       : "advanced",
    "specification"      : "kafka.2u4g.cluster.small * 3 broker",
    "status"             : "RUNNING",
    "storage_type"       : "hec",
    "user_id"            : "e4b27d49128e4bd0893b28d032a2e7c0",
    "user_name"          : "xxxx"
  },
  "fields": {
    "created_at"          : "1693203968959",
    "maintain_begin"      : "02:00:00",
    "maintain_end"        : "06:00:00",
    "storage_space"       : 186,
    "total_storage_space" : 300,
    "message"             : "{Instance JSON Data}"
  }
}

Note: Fields in tags, fields may change with subsequent updates

Note 1: The value of tags.name is the instance ID, used as the unique identifier

Note 2: The following fields are all JSON serialized strings

```