Skip to content

Alibaba Cloud KafKa

Alibaba Cloud KafKa includes instance disk usage, instance and topic message production volume, message production count, message consumption volume, message consumption count, etc. These metrics reflect the reliability guarantees of Kafka in handling large-scale message delivery and real-time data streams.

Configuration

Install Func

It is recommended to activate TrueWatch Integration - Extensions - DataFlux Func (Automata)

If deploying Func manually, refer to Manual Deployment of Func

Activate Script

Note: Please prepare the Alibaba Cloud AK that meets the requirements in advance (for simplicity, you can directly grant the global read-only permission ReadOnlyAccess)

Activate Script for Managed Version

  1. Log in to the TrueWatch console
  2. Click the 【Integration】 menu, select 【Cloud Account Management】
  3. Click 【Add Cloud Account】, select 【Alibaba Cloud】, and fill in the required information on the interface. If the cloud account information has been configured before, ignore this step
  4. Click 【Test】, and if the test is successful, click 【Save】. If the test fails, please check the relevant configuration information and test again
  5. In the 【Cloud Account Management】 list, you can see the added cloud account. Click the corresponding cloud account to enter the details page
  6. Click the 【Integration】 button on the cloud account details page. In the Not Installed list, find Alibaba Cloud KafKa, and click the 【Install】 button to pop up the installation interface and install it.

Activate Script Manually

  1. Log in to the Func console, click 【Script Market】, enter the TrueWatch script market, and search for: integration_alibabacloud_alikafka

  2. Click 【Install】, then enter the corresponding parameters: Alibaba Cloud AK ID, AK Secret, and account name.

  3. Click 【Deploy Startup Script】, the system will automatically create a Startup script set and automatically configure the corresponding startup script.

  4. After activation, you can see the corresponding automatic trigger configuration in 「Management / Automatic Trigger Configuration」. Click 【Execute】 to execute it immediately without waiting for the scheduled time. After a while, you can check the execution task records and corresponding logs.

Verification

  1. In 「Management / Automatic Trigger Configuration」, confirm whether the corresponding task has the corresponding automatic trigger configuration, and check the corresponding task records and logs to see if there are any exceptions
  2. In TrueWatch, check whether asset information exists in 「Infrastructure / Custom」
  3. In TrueWatch, check whether there is corresponding monitoring data in 「Metrics」

Metrics

After configuring Alibaba Cloud Cloud Monitor, the default Measurement is as follows. More metrics can be collected through configuration Alibaba Cloud Cloud Monitor Metric Details

Metric Id Metric Name Dimensions Statistics Unit
instance_disk_capacity_Maximum V2 Instance Disk Usage userId,instanceId Maximum %
instance_message_input Instance Message Production Volume userId,instanceId Value bytes/s
instance_message_num_input Instance Message Production Count userId,instanceId Value countSecond
instance_message_output Instance Message Consumption Volume userId,instanceId Value bytes/s
instance_reqs_input Instance Message Send Count userId,instanceId Value countSecond
instance_reqs_output Instance Message Consumption Count userId,instanceId Value countSecond
topic_message_input Topic Message Production Volume userId,instanceId Value bytes/s
topic_message_num_input Topic Message Production Count userId,instanceId Value countSecond
topic_message_output Topic Message Consumption Volume userId,instanceId Value bytes/s
topic_reqs_input Topic Message Send Count userId,instanceId Value countSecond
topic_reqs_output Topic Message Consumption Count userId,instanceId Value countSecond

Objects

The object data structure collected from Alibaba Cloud KafKa can be seen in 「Infrastructure - Custom」

{
  "measurement": "aliyun_kafka",
  "tags": {
    "InstanceId"        : "alikafka_post-cn-zsk3cluq100d",
    "InstanceName"      : "alikafka_post-cn-zsk3cluq100d",
    "RegionId"          : "cn-hangzhou",
    "ResourceGroupId"   : "rg-acfmv3ro3xnfwaa",
    "SpecType"          : "normal",
    "cloud_provider"    : "aliyun",
    "name"              : "alikafka_post-cn-zsk3cluq100d"
  },
  "fields": {
    "AllConfig"         : "{}",
    "CreateTime"        : 1692080715000,
    "DeployType"        : 5,
    "EipMax"            : 0,
    "ExpiredTime"       : 2007699914000,
    "IoMax"             : 20,
    "IoMaxSpec"         : "alikafka.hw.2xlarge",
    "PaidType"          : 1,
    "ServiceStatus"     : 0,
    "TopicNumLimit"     : 1000,
    "UsedGroupCount"    : 0,
    "UsedTopicCount"    : 0,
  }
}