Collector "Huawei Cloud - Cloud Monitoring" Configuration Manual¶

Before reading this document, please read:

• Cloud Integration

1. Configuration Structure¶

The configuration structure of this collector is as follows:

2. Configuration Examples¶

Specify Specific Metrics¶

Collect the 2 metrics named capacity_total and capacity_archive in SYS.OBS.

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['capacity_total', 'capacity_archive']
        }
    ]
}

Wildcard Matching Metrics¶

Metric names can be matched using the * wildcard.

In this example, the following metrics will be collected:

• Metrics named capacity_total
• Metrics starting with capacity
• Metrics ending with total
• Metrics containing capacity

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['capacity_total', 'capacity*', '*total', '*capacity*']
        }
    ]
}

Exclude Specific Metrics¶

Adding the "NOT" tag at the beginning indicates that the following metrics will be excluded.

In this example, the following metrics will [not] be collected:

• Metrics named capacity_total
• Metrics starting with capacity
• Metrics ending with total
• Metrics containing capacity

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['NOT', 'capacity_total', 'capacity*', '*total', '*capacity*']
        }
    ]
}

Multiple Filters to Specify Required Metrics¶

The same namespace can be specified multiple times, and the metrics will be filtered sequentially from top to bottom.

In this example, the following filtering steps are performed on the metric names:

1. Select all metrics containing capacity in their names.
2. From the results of the previous step, exclude metrics named capacity_total.

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['*capacity*']
        },
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['NOT', 'capacity_total']
        }
    ]
}

Configure Filters (Optional)¶

This collector script supports user-defined filters to allow users to filter target resources based on object attributes. The filter function returns True or False.

• True: The target resource should be collected.
• False: The target resource should not be collected.

Huawei Cloud Cloud Monitoring supports filtering object attributes that are consistent with the tags of cloud servers (ECS), cloud databases (RDS, Redis), load balancers (ELB), and object storage (OBS) custom objects:

# Example: Enable the filter to filter based on the instance_id and name attributes of the object. The configuration format is as follows:
def filter_instance(instance, namespace='A'):
    '''
    Collect metrics with namespace A and id xxxx
    '''
    # return True
    instance_id = instance['tags'].get('id')
    if instance_id in ['xxx']:
        return True
    return False


###### Do not modify the following contents #####
from integration_core__runner import Runner
import integration_huaweicloud_ces__main as main


@DFF.API('HuaweiCloud-Cloud monitoring and acquisition', timeout=3600, fixed_crontab='*/5 * * * *')
def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

3. Data Collection Instructions¶

Cloud Product Configuration Information¶

Cache Mechanism¶

Huawei Cloud Cloud Monitoring introduces a cache mechanism when obtaining metric dimension information. If users frequently create and delete instances, they need to wait for 1 hour before the collector starts collecting new monitoring data.

4. Data Reporting Format¶

After data is successfully synchronized, you can view the data in the "Metrics" section of TrueWatch.

Take the following collector configuration as an example:

huaweicloud_ces_configs = {
    'regions': ['cn-north-4'],
    'targets': [
        {
            'namespace': 'SYS.OBS',
            'metrics'  : ['capacity_total']
        }
    ]
}

The reported data example is as follows:

{
  "measurement": "huaweicloud_SYS.OBS",
  "tags": {
    "bucket_name": "i-xxx"
  },
  "fields": {
    "capacity_total_average" : "{...}",
    "capacity_total_max"     : "{...}",
    "capacity_total_min"     : "{...}",
    "capacity_total_sum"     : "{...}",
    "capacity_total_variance": "{...}"
  }
}

5. Interaction with Custom Object Collectors¶

When other custom object collectors (e.g., OBS) are running in the same DataFlux Func, this collector will supplement fields based on the dimension information in the Data Collection Instructions table. For example, OBS attempts to match the bucket_name field returned by cloud monitoring data with the tags.name field in the custom object.

Since it is necessary to know the custom object information in advance for interaction in the cloud monitoring collector, it is generally recommended to place the cloud monitoring collector at the end of the list, such as:

# Create collectors
collectors = [
    huaweicloud_obs.DataCollector(account, common_huaweicloud_configs),
    huaweicloud_ces.DataCollector(account, huaweicloud_ces_configs) # Cloud monitoring collector is usually placed last
]

When a successful match is made, additional fields from the tags of the matched custom object will be added to the tags of the cloud monitoring data, thereby achieving effects such as filtering cloud monitoring metric data using instance names. The specific effect is as follows:

Assume the original data collected by cloud monitoring is as follows:

{
  "measurement": "huaweicloud_SYS.OBS",
  "tags": {
    "bucket_name": "i-xxx"
  },
  "fields": { 
    "key": "value"
   }
}

At the same time, the custom object data collected by the Huawei Cloud OBS collector is as follows:

{
  "measurement": "huaweicloud_cvm",
  "tags": {
    "name"           : "xxx",
    "bucket_type"    : "xxx",
    "PlatformDetails": "xxx",
    "{...}"          : "{...}"
  },
  "fields": { 
    "key": "value" 
  }
}

Then, the final reported cloud monitoring data is as follows:

{
  "measurement": "huaweicloud_SYS.OBS",
  "tags": {
    "name"            : "xxx",
    "bucket_name"     : "xxx", // Original field from cloud monitoring
    "bucket_type"     : "xxx", // Field from custom object OBS
    "PlatformDetails" : "xxx", // Field from custom object OBS
    "{...}"
  },
  "fields": { 
    "key": "value"
   }
}

6. Troubleshooting¶

When running the program, you may encounter the following error:

HTTPClientError: An HTTP Client raised an unhandled exception: SoftTimeLimitExceeded()

Reason: The task execution time is too long, causing a timeout.

Solution:

• Reduce the number of collected metrics, clarify requirements, and only collect the metrics you actually need.
• Appropriately increase the timeout setting of the startup function, such as:

# Set the timeout to 120 seconds
@DFF.API('Execute collection', timeout=120)
def run():
    # Specific code omitted
    pass

7. Cloud Monitoring Call Count Instructions¶

X. Appendix¶

Huawei Cloud Cloud Monitoring¶

Please refer to the official Huawei Cloud documentation:

• Region List
• Huawei Cloud Namespaces
• Get Project ID