Skip to content

AWS Kinesis Analytics

Use the "TrueWatch Cloud Sync" script packages in the script market to synchronize cloud monitoring and cloud asset data to TrueWatch

Configuration

Install Func

We recommend enabling hosted Func under Integration - Extensions in TrueWatch. All prerequisites are installed automatically; continue with the script installation.

If deploying Func manually, refer to Manual Deployment of Func

Install Script

Note: Please prepare the required Amazon AK in advance (for simplicity, you can grant global read-only permissions ReadOnlyAccess)

Activate Scripts in Hosted Func

  1. Log in to the TrueWatch console
  2. Click the 【Integration】 menu and select 【Cloud Account Management】
  3. Click 【Add Cloud Account】, select 【AWS】, and fill in the required information on the interface. If you have already configured the cloud account information before, ignore this step
  4. Click 【Test】, and if the test is successful, click 【Save】. If the test fails, please check if the relevant configuration information is correct and test again
  5. Click 【Cloud Account Management】, and you can see the added cloud account in the list. Click the corresponding cloud account to enter the details page
  6. Click 【Integration】 on the cloud account details page, find AWS Kinesis Analytics in the Not Installed list and click 【Install】 to open the installation form.

Manual Activation Script

  1. Log in to the Func console, click 【Script Market】, enter the TrueWatch script market, and search for: integration_aws_kinesis_analytics

  2. Click 【Install】, then enter the corresponding parameters: AWS AK ID, AK Secret, and account name.

  3. Click 【Deploy Startup Script】 to create the Startup script set and configure the corresponding startup scripts automatically.

  4. After enabling the script, open 【Management / Automatic Trigger Configuration】 to view its automatic trigger configuration. Click 【Execute】 to run it immediately without waiting for the scheduled time. After a short wait, review the task records and logs.

Verification

  1. Under 【Management / Automatic Trigger Configuration】, confirm that the task has an automatic trigger configuration. Review its task records and logs for errors.
  2. In TrueWatch, check [Infrastructure / Custom] for asset information.
  3. In TrueWatch, check 【Metrics】 for the corresponding monitoring data.

Metrics

After configuring Amazon Cloud Monitoring, the default measurement sets are as follows. More metrics can be collected through configuration. Amazon Cloud Monitoring Metrics Details

Metric Description Unit
cpuUtilization The overall percentage of CPU utilization in the task manager. For example, if there are five task managers, Kinesis Data Analytics will publish five samples of this metric for each reporting interval. Percentage
containerCPUUtilization The overall percentage of CPU utilization in the task manager containers in the Flink application cluster. For example, if there are five task managers, there are correspondingly five TaskManager containers, and Kinesis Data Analytics publishes 2 * 5 samples of this metric for each 1-minute reporting interval. Percentage
containerMemoryUtilization The overall percentage of memory utilization in the task manager containers in the Flink application cluster. For example, if there are five task managers, there are correspondingly five TaskManager containers, and Kinesis Data Analytics publishes 2 * 5 samples of this metric for each 1-minute reporting interval. Percentage
containerDiskUtilization The overall percentage of disk utilization in the task manager containers in the Flink application cluster. For example, if there are five task managers, there are correspondingly five TaskManager containers, and Kinesis Data Analytics publishes 2 * 5 samples of this metric for each 1-minute reporting interval. Percentage
heapMemoryUtilization The overall heap memory utilization of the task managers. For example, if there are five task managers, Kinesis Data Analytics will publish five samples of this metric for each reporting interval. Percentage
oldGenerationGCCount The total number of old garbage collection operations performed in all task managers. Count
oldGenerationGCTime The total time spent executing old garbage collection operations. Milliseconds
threadCount The total number of live threads used by the application. Count

Object

The collected AWS KinesisAnalytics object data structure can be viewed in 「Infrastructure - Custom」

{
  "measurement": "aws_kinesis_analytics",
  "tags": {
    "ApplicationARN": "arn:aws-cn:xxxx:xxxx",
    "ApplicationMode": "STREAMING",
    "ApplicationName": "zsh_test",
    "ApplicationStatus": "RUNNING",
    "ApplicationVersionId": "3",
    "RegionId": "cn-northwest-1",
    "RuntimeEnvironment": "FLINK-1_15",
    "name": "zsh_test"
  },
  "fields": {
    "message"     : "{Instance JSON Data}"
  }
}

Note: The fields in tags and fields may change with subsequent updates