Collector "Alibaba Cloud - CloudMonitor" Configuration Manual¶

Before reading this document, please read:

• Cloud Integration

1. Configuration Structure¶

The configuration structure of this collector is as follows:

2. Configuration Examples¶

Specify Specific Metrics¶

Collect 2 metrics named CPUUtilization and concurrentConnections from ECS.

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['CPUUtilization', 'concurrentConnections'],
        },
    ],
}

Wildcard Matching for Metrics¶

Metric names can be matched using the * wildcard.

In this example, the following metrics will be collected:

• Metrics named CPUUtilization
• Metrics whose names start with CPU
• Metrics whose names end with Connections
• Metrics whose names contain Conn

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['CPUUtilization', 'CPU*', '*Connections', '*Conn*'],
        },
    ],
}

Exclude Specific Metrics¶

Adding the "NOT" tag at the beginning indicates that the following metrics will be excluded.

In this example, the following metrics will [not] be collected:

• Metrics named CPUUtilization
• Metrics whose names start with CPU
• Metrics whose names end with Connections
• Metrics whose names contain Conn

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['NOT', 'CPUUtilization', 'CPU*', '*Connections', '*Conn*'],
        },
    ],
}

Multiple Filters to Specify Required Metrics¶

The same namespace can be specified multiple times, filtering the metric names sequentially from top to bottom.

In this example, the metric names are filtered as follows:

1. Select all metrics whose names contain CPU
2. From the results of the previous step, exclude metrics named CPUUtilization

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['*CPU*'],
        },
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['NOT', 'CPUUtilization'],
        },
    ],
}

Configure Filters (Optional)¶

This collector script supports custom filters, allowing users to filter target resources based on object attributes. The filter function returns True｜False

• True: The target resource should be collected.
• False: The target resource should not be collected.

Supported object attributes for filtering:

When custom object collection is enabled, more object attributes can be filtered. For details, refer to the corresponding product's custom object collector documentation (coming soon...).

# Example: Enable filter to filter based on the InstanceId and RegionId attributes of the object. The configuration format is as follows:

def filter_instance(instance, namespace='acs_ecs_dashboard'):
    '''
    Collect metrics for instances with InstanceId i-xxxxxa, i-xxxxxb and RegionId cn-hangzhou
    '''
    instance_id = instance['tags'].get('InstanceId')
    region_id = instance['tags'].get('RegionId')
    if instance_id in ['i-xxxxxa', 'i-xxxxxb'] and region_id in ['cn-hangzhou']:
        return True
    return False

from integration_core__runner import Runner
import integration_alibabacloud_monitor__main as main

@DFF.API('AlibabaCloud-monitor ', timeout=3600, fixed_crontab="*/5 * * * *")
def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

3. Data Reporting Format¶

After data is successfully synchronized, you can view the data in the "Metrics" section of TrueWatch.

Take the following collector configuration as an example:

collector_configs = {
    'targets': [
        {
            'namespace': 'acs_ecs_dashboard',
            'metrics'  : ['CPUUtilization'],
        },
    ],
}

The reported data example is as follows:

{
  "measurement": "aliyun_acs_ecs_dashboard",
  "tags": {
    "instanceId": "i-xxxxx",
    "userId"    : "xxxxx"
  },
  "fields": {
    "CPUUtilization_Average": 1.23,
    "CPUUtilization_Maximum": 1.23,
    "CPUUtilization_Minimum": 1.23
  }
}

4. Integration with Custom Object Collectors¶

When other custom object collectors (such as ECS, RDS) are running in the same DataFlux Func, this collector will automatically try to match the tags.instanceId field with the tags.name field in the custom objects.

Since custom object information needs to be obtained first for integration in CloudMonitor-type collectors, it is generally recommended to place the CloudMonitor collector at the end of the list, for example:

    # Create collectors
    collectors = [
        aliyun_ecs.DataCollector(account, common_aliyun_configs),
        aliyun_rds.DataCollector(account, common_aliyun_configs),
        aliyun_slb.DataCollector(account, common_aliyun_configs),
        aliyun_oss.DataCollector(account, common_aliyun_configs),
        aliyun_monitor.DataCollector(account, monitor_collector_configs), # CloudMonitor-type collectors are usually placed last.
    ]

When a successful match is made, all fields except name in the custom object tags will be added to the tags of the monitoring data, enabling effects such as filtering CloudMonitor metric data using instance names. The specific effect is as follows:

Assume the original data collected by CloudMonitor is as follows:

{
  "measurement": "aliyun_acs_ecs_dashboard",
  "tags": {
    "instanceId": "i-001",
    "{...}"     : "{...}"
  },
  "fields": {
    "{...}"     : "{...}"
  }
}

At the same time, the custom object data collected by the Alibaba Cloud ECS collector is as follows:

{
  "measurement": "aliyun_ecs",
  "tags": {
    "name"      : "i-001",
    "InstanceId": "i-001",
    "RegionId"  : "cn-hangzhou",
    "{...}"     : "{...}"
  },
  "fields": {
    "{...}"     : "{...}"
  }
}

Then, the final reported CloudMonitor data is as follows:

{
"measurement": "aliyun_acs_ecs_dashboard",
  "tags": {
    "instanceId": "i-001",
    "RegionId"  : "cn-hangzhou",
    "{...}"     : "{...}"
  },
  "fields": {
    "{...}"     : "{...}"
  }
}

5. CloudMonitor API Call Count Explanation¶

Alibaba Cloud CloudMonitor has free quota limits for some API calls (currently: 1 million queries per month for free, and any excess is charged at 0.12 yuan per 10,000 calls). The DescribeMetricLast API used by this collector is also within this limit. The following explains the script set call count in detail:

1. User has multiple resources and needs to collect multiple monitoring items. Will it exceed the free quota?

This collector uses DescribeMetricLast (query the latest monitoring data for specified monitoring items). One request can obtain the latest monitoring data for multiple (up to 1000, if more, paging is required) resources for one monitoring item. Examples of request counts:

• If there are 1000 ECS resources under the account and you need to collect 1 monitoring item, CPUUtilization, it requires 1 request;
• If there are 1000 ECS resources under the account and you need to collect 2 monitoring items, CPUUtilization and DiskReadBPS, it requires 2 requests (one request per monitoring item);
• If there are 1001 ECS resources under the account and you need to collect 1 monitoring item, CPUUtilization, it requires 2 requests (resources exceed 1000, paging is required);
• If there are 1001 ECS resources under the account and you need to collect 2 monitoring items, CPUUtilization and DiskReadBPS, it requires 4 requests;

2. View the actual call count in the task execution log:

The collector counts the number of API calls for each task execution result, which can be viewed in the log. Example:

[2023-04-21 15:32:13.194] [+0ms] The [1st] account collection is completed, taking [274 milliseconds], with [2] API calls during this period.
[2023-04-21 15:32:13.194] [+0ms] Detailed calls are as follows:
[2023-04-21 15:32:13.194] [+0ms] -> metrics.aliyuncs.com/?Action=DescribeMetricMetaList: 1 call
[2023-04-21 15:32:13.194] [+0ms] -> metrics.aliyuncs.com/?Action=DescribeMetricLast: 1 call

!!! warning "Given the free quota limits for CloudMonitor API calls, it is recommended that users configure monitoring items as needed to avoid additional costs caused by wildcard usage."

Notes¶

Common Errors and Solutions¶

1. The number of collected instances does not match the actual number of existing instances

Reason: The instance status is shutdown.

Solution:

• Start the instance.

X. Appendix¶

Please refer to the official Alibaba Cloud documentation:

• CloudMonitor / Appendix 1: Metrics
• CloudMonitor Basic / Free quotas
• CloudMonitor Basic / Pay-as-you-go