Skip to content

AWS KinesisAnalytics

Use the "Cloud Sync" series of script packages in the script market to synchronize cloud monitoring and cloud asset data to TrueWatch

Configuration

Install Func

It is recommended to activate TrueWatch Integration - Extensions - DataFlux Func (Automata): All prerequisites are automatically installed. Please proceed with the script installation.

If deploying Func manually, refer to Manual Deployment of Func

Install Script

Note: Please prepare the required Amazon AK in advance (for simplicity, you can directly grant the global read-only permission ReadOnlyAccess)

Activate Script in Automata

  1. Log in to the TrueWatch console
  2. Click the 【Integration】 menu and select 【Cloud Account Management】
  3. Click 【Add Cloud Account】, select 【AWS】, and fill in the required information on the interface. If you have already configured the cloud account information, ignore this step.
  4. Click 【Test】, and if the test is successful, click 【Save】. If the test fails, please check if the relevant configuration information is correct and retest.
  5. Click 【Cloud Account Management】, and you can see the added cloud account in the list. Click the corresponding cloud account to enter the details page.
  6. Click the 【Integration】 button on the cloud account details page, find AWS KinesisAnalytics under the Not Installed list, and click the 【Install】 button. Follow the installation interface to complete the installation.

Manual Script Activation

  1. Log in to the Func console, click 【Script Market】, enter the TrueWatch script market, and search for: integration_aws_kinesis_analytics

  2. Click 【Install】, then enter the corresponding parameters: AWS AK ID, AK Secret, and account name.

  3. Click 【Deploy Startup Script】, the system will automatically create the Startup script set and configure the corresponding startup script.

  4. After activation, you can see the corresponding automatic trigger configuration in 「Manage / Automatic Trigger Configuration」. Click 【Execute】 to immediately execute it once without waiting for the scheduled time. Wait a moment, and you can view the execution task records and corresponding logs.

Verification

  1. Confirm in 「Manage / Automatic Trigger Configuration」 whether the corresponding task has the automatic trigger configuration. You can also check the corresponding task records and logs for any exceptions.
  2. In TrueWatch, check if the asset information exists in 「Infrastructure / Custom」.
  3. In TrueWatch, check if the corresponding monitoring data exists in 「Metrics」.

Metrics

After configuring Amazon Cloud Monitoring, the default measurement sets are as follows. More metrics can be collected through configuration. Amazon Cloud Monitoring Metrics Details

Metric Description Unit
cpuUtilization The overall percentage of CPU utilization in the task manager. For example, if there are five task managers, Kinesis Data Analytics will publish five samples of this metric for each reporting interval. Percentage
containerCPUUtilization The overall percentage of CPU utilization in the task manager containers in the Flink application cluster. For example, if there are five task managers, there are correspondingly five TaskManager containers, and Kinesis Data Analytics publishes 2 * 5 samples of this metric for each 1-minute reporting interval. Percentage
containerMemoryUtilization The overall percentage of memory utilization in the task manager containers in the Flink application cluster. For example, if there are five task managers, there are correspondingly five TaskManager containers, and Kinesis Data Analytics publishes 2 * 5 samples of this metric for each 1-minute reporting interval. Percentage
containerDiskUtilization The overall percentage of disk utilization in the task manager containers in the Flink application cluster. For example, if there are five task managers, there are correspondingly five TaskManager containers, and Kinesis Data Analytics publishes 2 * 5 samples of this metric for each 1-minute reporting interval. Percentage
heapMemoryUtilization The overall heap memory utilization of the task managers. For example, if there are five task managers, Kinesis Data Analytics will publish five samples of this metric for each reporting interval. Percentage
oldGenerationGCCount The total number of old garbage collection operations performed in all task managers. Count
oldGenerationGCTime The total time spent executing old garbage collection operations. Milliseconds
threadCount The total number of live threads used by the application. Count

Objects

The collected AWS KinesisAnalytics object data structure can be viewed in 「Infrastructure - Custom」

{
  "measurement": "aws_kinesis_analytics",
  "tags": {
    "ApplicationARN": "arn:aws-cn:xxxx:xxxx",
    "ApplicationMode": "STREAMING",
    "ApplicationName": "zsh_test",
    "ApplicationStatus": "RUNNING",
    "ApplicationVersionId": "3",
    "RegionId": "cn-northwest-1",
    "RuntimeEnvironment": "FLINK-1_15",
    "name": "zsh_test"
  },
  "fields": {
    "message"     : "{Instance JSON Data}"
  }
}

Note: The fields in tags and fields may change with subsequent updates ```