Collector 'AWS-Kinesis' Configuration Manual¶

Before reading this document, please read:

Cloud Integration

Tip

Before using this collector, you must install the 'Integration Core Package' and its corresponding third-party dependencies.

1. Configuration Structure¶

The configuration structure of this collector is as follows:

Field	Type	Required	Description
`regions`	list	Required	List of regions to be collected
`regions[#]`	str	Required	Region ID. For example: `'cn-north-1'` See appendix for the complete list

2. Configuration Example¶

Collect instance data from the Beijing region

collector_configs = {
    'regions': ['cn-north-1' ]
}

Configuration Filter (Optional)¶

This collector script supports custom filters, allowing users to filter target resources based on object attributes. The filter function returns True｜False

True: The target resource needs to be collected.
False: The target resource does not need to be collected.

# Example: Enable the filter, filter based on the StreamName attribute of the object, the configuration format is as follows:
def filter_instance(instance):

    # return True
    if instance in ['xxx'] or instance in ['yyy']:
        return True
    return False


@DFF.API('AWS-Kinesis Collection', timeout=3600, fixed_crontab='* * * * *')
def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

3. Data Reporting Format¶

After data is synchronized normally, you can view the data in the 'Infrastructure-Resource Catalog' of TrueWatch.

An example of the reported data is as follows:

{
  "measurement": "aws_kinesis",
  "tags": {
    "EncryptionType"      : "NONE",
    "HasMoreShards"       : "false",
    "RetentionPeriodHours": "24",
    "StreamARN"           : "arn:xxx1",
    "StreamName"          : "zsh_test",
    "StreamStatus"        : "ACTIVE",
    "name"                : "zsh_test"
  },
  "fields": {
    "Shards" : "{Stream Shard Information}",
    "message": "{Instance JSON Data}"
  }
}

Note

Fields in tags and fields may change with subsequent updates.

4. IAM Policy Permissions¶

Note

If users use the method of bringing IAM roles to collect resources, certain operation permissions need to be enabled.

This collector requires the following permissions:

kinesis:ListStreams

kinesis:DescribeStream

X. Appendix¶

Please refer to the AWS official documentation: