Alibaba Cloud ElasticSearch¶
Alibaba Cloud ElasticSearch Metrics display, including cluster status, index QPS, node CPU/memory/disk usage, etc.
Configuration¶
Install Func¶
It is recommended to activate TrueWatch Integration - Extensions - DataFlux Func (Automata)
If you want to deploy Func by yourself, refer to Deploy Func by Yourself
Activate Script¶
Note: Please prepare the Alibaba Cloud AK with the required permissions in advance (for simplicity, you can grant the global read-only permission
ReadOnlyAccess
)
Activate Script for DataFlux Func (Automata)¶
- Log in to the TrueWatch console
- Click the 【Integration】 menu, and select 【Cloud Account Management】
- Click 【Add Cloud Account】, select 【Alibaba Cloud】, and fill in the required information on the interface. If you have already configured the cloud account information before, ignore this step
- Click 【Test】, and click 【Save】 after the test is successful. If the test fails, please check whether the relevant configuration information is correct and test again
- Click 【Cloud Account Management】, and you can see the added cloud account in the list. Click the corresponding cloud account to enter the details page
- Click the 【Integration】 button on the cloud account details page, find
Alibaba Cloud ElasticSearch
under theNot Installed
list, and click the 【Install】 button to pop up the installation interface for installation.
Activate Script Manually¶
-
Log in to the Func console, click 【Script Market】, enter the TrueWatch script market, and search:
integration_alibabacloud_elasticsearch
-
Click 【Install】, and enter the corresponding parameters: Alibaba Cloud AK ID, AK Secret, and account name.
-
Click 【Deploy Startup Script】, the system will automatically create the
Startup
script set, and automatically configure the corresponding startup script. -
After activation, you can see the corresponding automatic trigger configuration in 「Management / Automatic Trigger Configuration」. Click 【Execute】 to execute it immediately without waiting for the scheduled time. After a while, you can check the execution task records and corresponding logs.
Verification¶
- In 「Management / Automatic Trigger Configuration」, confirm whether the corresponding task has the corresponding automatic trigger configuration, and check the corresponding task records and logs to check for any exceptions
- In TrueWatch, check whether there is asset information in 「Infrastructure / Custom」
- In TrueWatch, check whether there is corresponding monitoring data in 「Metrics」
Metrics¶
After configuring Alibaba Cloud Cloud Monitor, the default Measurement is as follows. You can collect more Metrics through configuration Alibaba Cloud Cloud Monitor Metrics Details
Metric Id | Metric Name | Dimensions | Statistics | Unit |
---|---|---|---|---|
ClusterAutoSnapshotLatestStatus |
Snapshot Status | userId,clusterId | Maximum | value |
ClusterIndexQPS |
Cluster Write QPS | userId,clusterId | Average | Count/Second |
ClusterQueryQPS |
Cluster Query QPS | userId,clusterId | Average | Count/Second |
ClusterStatus |
Cluster Status | userId,clusterId | Value,Maximum | value |
NodeCPUUtilization |
Elasticsearch Instance Node CPU Usage | userId,clusterId,nodeIP | Average,Maximum | % |
NodeDiskUtilization |
Elasticsearch Instance Node Disk Usage | userId,clusterId,nodeIP | Average,Maximum | % |
NodeHeapMemoryUtilization |
Elasticsearch Instance Node HeapMemory Usage | userId,clusterId,nodeIP | Average,Maximum | % |
NodeLoad_1m |
Node Load_1m | userId,clusterId,nodeIP | Average | value |
NodeStatsDataDiskR |
Number of Read Requests Completed per Second | userId,clusterId,nodeIP | Maximum | count |
NodeStatsDataDiskRm |
Size Read per Second | userId,clusterId,nodeIP | Maximum | MB/s |
NodeStatsDataDiskUtil |
IOUtil | userId,clusterId,nodeIP | Maximum | % |
NodeStatsDataDiskW |
Number of Write Requests Completed per Second | userId,clusterId,nodeIP | Maximum | count |
NodeStatsDataDiskWm |
Size Written per Second | userId,clusterId,nodeIP | Maximum | MB/s |
NodeStatsExceptionLogCount |
Exception Count | userId,clusterId,nodeIP | Maximum | Count |
NodeStatsFullGcCollectionCount |
FullGc Count | userId,clusterId,nodeIP | Maximum | Count |
NodeStatsNetworkinPackages |
Node Network Inbound Packages | userId,clusterId,nodeIP | Maximum | count |
NodeStatsNetworkinRate |
Data Inbound Rate | userId,clusterId,nodeIP | Maximum | kB/s |
NodeStatsNetworkoutPackages |
Node Network Outbound Packages | userId,clusterId,nodeIP | Maximum | count |
NodeStatsNetworkoutRate |
Data Outbound Rate | userId,clusterId,nodeIP | Maximum | kB/s |
NodeStatsTcpEstablished |
Node TCP Connections | userId,clusterId,nodeIP | Maximum | count |
Object¶
The collected Alibaba Cloud ElasticSearch object data structure can be seen in 「Infrastructure - Custom」
{
"measurement": "aliyun_elasticsearch",
"tags": {
"RegionId" : "cn-hangzhou",
"esVersion" : "7.4.0_with_X-Pack",
"instanceId" : "es-cn-xxxx",
"name" : "es-cn-xxxx",
"paymentType" : "prepaid",
"resourceGroupId": "rg-acfm2l3p7xxxx",
"serviceVpc" : "True",
"status" : "active"
},
"fields": {
"advancedDedicateMaster": false,
"createdAt" : "2021-04-07T06:10:50.527Z",
"extendConfigs" : "[ {Cluster Extension Parameter Configuration JSON Data}, ...]",
"message" : "{Instance JSON Data}"
}
}
Logging¶
Prerequisites¶
Note 1: Before using this collector, you must install the 「TrueWatch Integration Core Package」 and its supporting third-party dependencies Note 2: The code execution of this script depends on the mongodb instance object collection. If the custom object collection of mongodb is not configured, the slow log script cannot collect slow log data
Install Script¶
On the previous basis, you need to install a corresponding elasticsearch log collection script
In 「Management / Script Market」, click and install the corresponding script package:
- 「TrueWatch Integration (Alibaba Cloud - ElasticSearch Collection)」(ID:
integration_alibabacloud_elasticsearch_log
)
Data Reporting Format¶
After the data is synchronized normally, you can view the data in TrueWatch's 「Infrastructure - Custom Objects」.
The reported data example is as follows:
{
"measurement": "aliyun_elasticsearch_log",
"tags": {
"RegionId" : "cn-hangzhou",
"esVersion" : "7.10.0_with_X-Pack",
"host" : "10.14.xxx.xxx",
"instanceId" : "es-cn-xxxx",
"name" : "es-cn-xxxx",
"paymentType" : "prepaid",
"resourceGroupId": "rg-aekzkcwe4dxxxx",
"serviceVpc" : "True",
"status" : "active"
},
"fields": {
"timestamp" : 1684304299000,
"contentCollection": "[ {Log Detail Information JSON Data}, ...]",
"message" : "{Instance JSON Data}"
}
}
log_types (log type) assignment meaning:
Value | Description |
---|---|
INSTANCELOG |
Main Log |
SEARCHSLOW |
Searching Slow Log |
INDEXINGSLOW |
Indexing Slow Log |
JVMLOG |
GC Log |
ES_SEARCH_ACCESS_LOG |
ES Access Log |
AUDIT |
Audit Log |
Note: The fields in
tags
andfields
may change with subsequent updates Note 1: The value of tags.name is the instance ID, which is used as the unique identifier Note 2:fields.message
is a JSON serialized string ```