Huawei Cloud DMS Kafka

Collect Huawei Cloud DMS Kafka Metrics

Configuration¶

Install Func¶

It is recommended to activate TrueWatch Integration - Extensions - DataFlux Func (Automata): All prerequisites are automatically installed, please proceed with the script installation.

If you need to deploy Func manually, refer to Manual Deployment of Func

Install Script¶

Note: Please prepare the Huawei Cloud AK in advance (for simplicity, you can grant the global read-only permission ReadOnlyAccess)

1. Log in to the Func console, click 【Script Market】, enter the TrueWatch Script Market, and search for ID: integration_huaweicloud_kafka

2. Click 【Install】, then enter the corresponding parameters: Huawei Cloud AK, SK, and account name

3. Click 【Deploy Startup Script】, the system will automatically create the Startup script set and configure the corresponding startup script

4. After enabling, you can see the corresponding automatic trigger configuration in 「Manage / Automatic Trigger Configuration」. Click 【Execute】 to immediately execute once without waiting for the scheduled time. Wait a moment, you can view the execution task records and corresponding logs

Verification¶

1. In 「Manage / Automatic Trigger Configuration」, confirm whether the corresponding task has the automatic trigger configuration, and check the corresponding task records and logs for any exceptions
2. In TrueWatch, check if there is asset information in 「Infrastructure - Resource Catalog」
3. In TrueWatch, check if there is corresponding monitoring data in 「Metrics」

Metrics¶

Collect Huawei Cloud DMS Kafka Metrics, more metrics can be collected through configuration Huawei Cloud DMS Kafka Metrics Details

Instance Monitoring Metrics¶

Node Monitoring Metrics¶

Consumer Group Monitoring Metrics¶

Object¶

The collected Huawei Cloud Kafka object data structure can be seen in 「Infrastructure - Resource Catalog」

{
  "measurement": "huaweicloud_kafka",
  "tags": {      
    "RegionId"           : "cn-north-4",
    "charging_mode"      : "1",
    "connect_address"    : "192.168.0.161,192.168.0.126,192.168.0.31",
    "description"        : "",
    "engine"             : "kafka",
    "engine_version"     : "2.7",
    "instance_id"        : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "kafka_manager_user" : "",
    "name"               : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "port"               : "9092",
    "project_id"         : "f5f4c067d68xxxx86e173b18367bf",
    "resource_spec_code" : "",
    "service_type"       : "advanced",
    "specification"      : "kafka.2u4g.cluster.small * 3 broker",
    "status"             : "RUNNING",
    "storage_type"       : "hec",
    "user_id"            : "e4b27d49128e4bd0893b28d032a2e7c0",
    "user_name"          : "xxxx"
  },
  "fields": {
    "created_at"          : "1693203968959",
    "maintain_begin"      : "02:00:00",
    "maintain_end"        : "06:00:00",
    "storage_space"       : 186,
    "total_storage_space" : 300,
    "message"             : "{Instance JSON Data}"
  }
}

Note: The fields in tags and fields may change with subsequent updates

Tip 1: The value of tags.name is the instance ID, used as the unique identifier

Tip 2: The following fields are all JSON serialized strings