Volcengine ESCloud Cloud Search Service
Collect Metrics data from Volcengine ESCloud Cloud Search Service
Configuration¶
Install Func¶
It is recommended to enable TrueWatch Integration - Extensions - DataFlux Func (Automata): All prerequisites are automatically installed, please proceed with the script installation.
If you want to deploy Func yourself, refer to Self-deploy Func
Install Script¶
Note: Please prepare the required Volcengine AK in advance (for simplicity, you can directly grant the global read-only permission
ReadOnlyAccess
)
To synchronize the monitoring data of ESCloud cloud resources, we install the corresponding collection script: 「TrueWatch Integration (Volcengine - Cloud Search Service Collection)」(ID: integration_volcengine_es_cloud
)
Click 【Install】, then enter the corresponding parameters: Volcengine AK, Volcengine account name, and Regions.
Click 【Deploy Startup Script】, the system will automatically create the Startup
script set and configure the corresponding startup script.
After enabling, you can see the corresponding automatic trigger configuration in 「Manage / Automatic Trigger Configuration」. Click 【Execute】 to immediately execute it once without waiting for the scheduled time. Wait for a moment, and you can check the execution task records and corresponding logs.
Verification¶
- Confirm whether the corresponding task exists in the automatic trigger configuration in 「Manage / Automatic Trigger Configuration」, and check the task records and logs for any exceptions
- In TrueWatch, check if the asset information exists in 「Infrastructure - Resource Catalog」
- In TrueWatch, check if there is corresponding monitoring data in 「Metrics」
Metrics¶
Configure Volcengine ESCloud monitoring metrics to collect more metrics through configuration Volcengine Cloud Search Service Metrics Details
Volcengine ESCloud Cloud Search Service Monitoring Metrics¶
Volcengine ESCloud metrics are located under the volcengine_VCM_ESCloud
Measurement. Here are the related metric descriptions
MetricName |
Subnamespace |
Metric Name | MetricUnit | Dimension |
---|---|---|---|---|
ActivePrimaryShardNumber |
cluster |
Primary Shard Count | Count | ResourceID |
ActiveShardNumber |
cluster |
Active Shard Count | Count | ResourceID |
AvgDiskUsagePercent |
total |
Average Disk Usage Rate | Percent | ResourceID |
AvgJvmUsage |
total |
Average JVM Usage Rate | Percent | ResourceID |
DeletedDocumentRate |
operation |
Delete Document Rate | None | ResourceID |
DiskUsagePercent |
resource |
Node Disk Usage Rate | Percent | ResourceID,Node |
FielddataMemorySize |
cluster |
Field data Memory Usage | Bytes | ResourceID,Node |
FsIoTotalReadBytes |
resource |
Node Disk Read Bandwidth | Bytes/Second | ResourceID,Node |
FsIoTotalWriteBytes |
resource |
Node Disk Write Bandwidth | Bytes/Second | ResourceID,Node |
GreenIndexNumber |
cluster |
Green Status Index Count | Count | ResourceID |
IndexOps |
operation |
Index Operations OPS | None | ResourceID |
InitializingShardNumber |
cluster |
Initializing Shard Count | Count | ResourceID |
InstanceCapacityUsage |
total |
Instance Used Capacity | Percent | ResourceID |
InstanceDataNodeNumber |
total |
Instance Data Node Count | Count | ResourceID |
InstanceHealthState |
total |
Cluster Status | None | ResourceID |
InstanceIngestNodeNumber |
total |
Instance Ingest Role Node Count | Count | ResourceID |
InstanceMasterNodeNumber |
total |
Instance Master Role Node Count | Count | ResourceID |
InstanceMasterOnlyNodeNumber |
total |
Instance Dedicated Master Node Count | Count | ResourceID |
InstanceStorageUsedBytes |
total |
Instance Storage Usage Size | Bytes | ResourceID |
JvmUsage |
resource |
Node JVM Usage Rate | Percent | ResourceID,Node |
MaxCpuUsage |
total |
Maximum CPU Usage Rate | Percent | ResourceID |
MaxDiskUsagePercent |
total |
Maximum Disk Usage Rate | Percent | ResourceID |
MaxJvmUsage |
total |
Maximum JVM Usage Rate | Percent | ResourceID |
MeanCpuUsage |
total |
Average CPU Usage Rate | Percent | ResourceID |
MergeOps |
operation |
Merge Operations OPS | None | ResourceID |
NodeCpuUsage |
resource |
Node CPU Usage Rate | Percent | ResourceID,Node |
NodeDiskReadIOPS |
resource |
Node Disk Read IOPS | Count/Second | ResourceID,Node |
NodeFsIops |
resource |
Node Data Directory IOPS | None | ResourceID,Node |
NodeHttpConnections |
resource |
Node HTTP Connections | Count | ResourceID,Node |
NodeWriteIOPS |
resource |
Node Disk Write IOPS | Count/Second | ResourceID,Node |
OldGcCountPerMin |
resource |
Old Generation GC Count Per Minute | None | ResourceID,Node |
QueryOps |
operation |
Query Operations OPS | None | ResourceID |
ReadOnlyIndexNumber |
cluster |
Read-Only Index Count | Count | ResourceID |
RedIndexNumber |
total |
Red Status Index Count | None | ResourceID |
RelocatingShardNumber |
cluster |
Relocating Shard Count | None | ResourceID |
ReplicaShardNumber |
cluster |
Replica Shard Count | Count | ResourceID |
SearchLatency |
operation |
Query Latency | Second | ResourceID |
SearchSuccessRatio |
operation |
Search Success Rate | Percent | ResourceID |
BulkSuccessRatio |
operation |
Bulk Success Rate | Percent | ResourceID |
ThreadPoolFlushRejected |
cluster |
Rejected Flush Tasks Count | Count | ResourceID,Node |
ThreadPoolForceMergeQueue |
cluster |
Force Merge Tasks in Queue Count | Count | ResourceID,Node |
ThreadPoolForceMergeRejected |
cluster |
Rejected Force Merge Tasks Count | Count | ResourceID,Node |
TotalDocumentNumber |
total |
Total Document Count | Count | ResourceID |
TotalIndexNumber |
total |
Total Index Count | Count | ResourceID |
TotalPendingTasks |
cluster |
Total Pending Tasks Count | Count | ResourceID |
TotalShardNumber |
total |
Total Shard Count | Count | ResourceID |
UnsignedShardNumber |
cluster |
Unassigned Shard Count | Count | ResourceID |
YellowIndexNumber |
cluster |
Yellow Status Index Count | Count | ResourceID |
YoungGcCountPerMin |
resource |
Young Generation GC Count Per Minute | None | ResourceID,Node |
Objects¶
The collected Volcengine ESCloud object data structure can be seen in 「Infrastructure - Resource Catalog」
{
"measurement": "volcengine_es_cloud",
"tags": {
"InstanceId" : "o-0089apbhxxx",
"RegionId" : "cn-guangzhou",
"Status" : "Running",
"ESEip" : "118.145.xxx.xxx",
"UserId" : "2102598885",
"ESEipId" : "eip-11vuua67wc3r440yrhczl0xxx",
"KibanaEipId" : "'\'",
"KibanaPublicDomain" : "'\'",
"CerebroPublicDomain" : "'\'",
"KibanaPrivateDomain" : "https://visual-o-xxxxxapbhpvmd.escloud.ivolces.com:5601",
"CerebroPrivateDomain" : "'\'",
"CerebroEnabled" : "True",
"EnableESPublicNetwork" : "True",
"EnableESPrivateNetwork" : "vpc-13fpvlahzboxsxxxxxxxxxx",
"EnableKibanaPublicNetwork" : "True",
"EnableKibanaPrivateNetwork" : "True",
"EnableESPrivateDomainPublic" : "False",
"EnableKibanaPrivateDomainPublic" : "False",
},
"fileds": {
"TotalNodes": "4",
"ChargeEnabled": "True",
"DeletionProtection": "True",
"CreateTime": "2025-05-06T09:39:50+08:00",
"ExpireDate": "'\'",
"KibanaConfig": "'\'",
"ResourceTags": "[]",
"SubInstances": "[]",
"TransferInfo": "{}",
"MaintenanceDay": "[\"MONDAY\", \"TUESDAY\", \"WEDNESDAY\", \"THURSDAY\", \"FRIDAY\", \"SATURDAY\", \"SUNDAY\"",
"ESPublicEndpoint": "https://elasticsearch-o-xxxxxxapbhpvmd.escloud.volces.com:9200",
"ESPrivateEndpoint": "https://elasticsearch-o-xxxxxapbhpvmd.escloud.ivolces.com:9200",
"ESPublicIpWhitelist": "127.0.0.1",
"ESPrivateIpWhitelist": "0.0.0.0/0",
"InstanceConfiguration": "{Json data}",
"KibanaPublicIpWhitelist": "'\'",
"KibanaPrivateIpWhitelist": "0.0.0.0/0"
}
}