AWS S3¶

Use the "Cloud Sync" series of script packages in the script market to synchronize cloud monitoring and cloud asset data to TrueWatch.

Configuration¶

Install Func¶

It is recommended to enable TrueWatch Integration - Extensions - DataFlux Func (Automata): all preconditions are automatically installed, please proceed with the script installation.

If you deploy Func yourself, refer to Self-deploy Func

Install Script¶

Note: Please prepare the Amazon AK with the required permissions in advance (for simplicity, you can directly grant the global read-only permission ReadOnlyAccess)

Enable Script for Automata¶

1. Log in to the TrueWatch console
2. Click the 【Integration】 menu, select 【Cloud Account Management】
3. Click 【Add Cloud Account】, select 【AWS】, and fill in the required information on the interface. If you have already configured the cloud account information before, ignore this step
4. Click 【Test】, and click 【Save】 after the test is successful. If the test fails, please check whether the relevant configuration information is correct and test again
5. Click 【Cloud Account Management】, you can see the added cloud account in the list, click the corresponding cloud account to enter the details page
6. Click the 【Integration】 button on the cloud account details page, find AWS S3 under the Not Installed list, and click the 【Install】 button to pop up the installation interface for installation.

Manually Enable Script¶

1. Log in to the Func console, click 【Script Market】, enter the TrueWatch script market, search for: integration_aws_s3

2. Click 【Install】, and enter the corresponding parameters: AWS AK ID, AK Secret, and account name.

3. Click 【Deploy Startup Script】, the system will automatically create the Startup script set and automatically configure the corresponding startup script.

4. After enabling, you can see the corresponding automatic trigger configuration in 「Management / Automatic Trigger Configuration」. Click 【Execute】 to execute it immediately without waiting for the scheduled time. Wait a moment, you can view the execution task records and corresponding logs.

Verification¶

1. In 「Management / Automatic Trigger Configuration」, confirm whether the corresponding task has the corresponding automatic trigger configuration, and you can also check the corresponding task records and logs to check for any abnormalities
2. In TrueWatch, check whether the asset information exists in 「Infrastructure / Custom」
3. In TrueWatch, check whether there is corresponding monitoring data in 「Metrics」

Metrics¶

After configuring Amazon CloudWatch, the default measurement sets are as follows. You can collect more metrics by configuration Amazon CloudWatch Metrics Details

Amazon S3 Daily Storage Metrics for Buckets in CloudWatch¶

The AWS/S3 namespace includes the following daily storage metrics for buckets.

Amazon S3 CloudWatch Request Metrics in CloudWatch¶

The AWS/S3 namespace includes the following request metrics. These metrics include non-billable requests (if they are GET requests from COPY and Replication).

S3 Replication Metrics in CloudWatch¶

You can use S3 Replication Metrics to monitor the progress of replication by tracking pending bytes, pending operations, and replication latency. For more information, see Monitoring Progress Using Replication Metrics.

Note:

You can enable alerts for replication metrics in Amazon CloudWatch. When setting an alert for replication metrics, set the Missing data treatment field to Treat missing data as ignore (maintain the alarm state).

S3 Storage Lens Storage Statistics Management Tool Metrics in CloudWatch¶

You can publish S3 Storage Lens Storage Statistics Management Tool usage and activity metrics to Amazon CloudWatch to create a unified view of operational health in CloudWatch dashboards. S3 Storage Lens Storage Statistics Management Tool metrics are published to the AWS/S3/Storage-Lens namespace in CloudWatch. The CloudWatch publishing option is available for S3 Storage Lens Storage Statistics Management Tool dashboards that have been upgraded to advanced metrics and recommendations.

For a list of S3 Storage Lens Storage Statistics Management Tool metrics published to CloudWatch, see Amazon S3 Storage Lens Storage Statistics Management Tool Metrics Glossary. For a complete list of dimensions, see Dimensions.

S3 Object Lambda Request Metrics in CloudWatch¶

S3 Object Lambda includes the following request metrics.

Metrics for Amazon S3 on Outposts in CloudWatch¶

For a list of metrics used for S3 on Outposts in CloudWatch, see CloudWatch metrics.

Dimensions for Amazon S3 in CloudWatch¶

The following dimensions are used to filter Amazon S3 metrics.

Dimensions for S3 Storage Lens Storage Statistics Management Tool in CloudWatch¶

For a list of dimensions used to filter S3 Storage Lens Storage Statistics Management Tool metrics in CloudWatch, see Dimensions.

Dimensions for S3 Object Lambda Requests in CloudWatch¶

The following dimensions are used to filter data from Object Lambda access points.

Object¶

The AWS S3 object data structure collected can be seen in 「Infrastructure - Custom」

{
  "measurement": "aws_s3",
  "tags": {
    "name"              : "dataxxxx",
    "RegionId"          : "cn-northwest-1",
    "LocationConstraint": "cn-northwest-1",
    "Name"              : "dataxxxx"
  },
  "fields": {
    "CreationDate": "2022-03-09T06:13:31Z",
    "Grants"      : "{JSON data}",
    "message"     : "{instance JSON data}"
  }
}

Note: The fields in tags and fields may change with subsequent updates

Tip 1: The value of tags.name is the instance name, used as a unique identifier

Tip 2: fields.message is a JSON serialized string

Tip 3: fields.Grants is the bucket access control list