Skip to content

AWS MSK

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that allows you to build and run applications that use Apache Kafka to process streaming data.

Use the TrueWatch Cloud Sync script package in the script market to synchronize cloud monitoring and cloud asset data to TrueWatch

Configuration

Install Func

It is recommended to enable the TrueWatch Integration - Extensions - DataFlux Func (Automata): all prerequisites are automatically installed, please proceed with the script installation.

If you are deploying Func yourself, refer to Self-deploying Func

Install Script

Note: Please prepare the required Amazon AK in advance (for simplicity, you can grant global read-only permissions ReadOnlyAccess)

Enable Script in Automata

  1. Log in to the TrueWatch console
  2. Click the 【Integration】 menu and select 【Cloud Account Management】
  3. Click 【Add Cloud Account】, select 【AWS】, and fill in the required information on the interface. If you have already configured the cloud account information, skip this step.
  4. Click 【Test】, if the test is successful, click 【Save】. If the test fails, please check the relevant configuration information and test again.
  5. Click 【Cloud Account Management】, you can see the added cloud account in the list. Click the corresponding cloud account to enter the details page.
  6. Click the 【Integration】 button on the cloud account details page, find AWS MSK under the Not Installed list, and click the 【Install】 button to pop up the installation interface and install it.

Manually Enable Script

  1. Log in to the Func console, click 【Script Market】, enter the TrueWatch script market, and search for integration_aws_kafka.

  2. Click 【Install】, then enter the corresponding parameters: AWS AK ID, AK Secret, and account name.

  3. Click 【Deploy Startup Script】, the system will automatically create the Startup script set and configure the corresponding startup scripts.

  4. After enabling, you can see the corresponding automatic trigger configuration in 「Manage / Automatic Trigger Configuration」. Click 【Execute】 to execute it immediately without waiting for the scheduled time. After a while, you can check the execution task records and corresponding logs.

Verification

  1. In 「Manage / Automatic Trigger Configuration」, confirm whether the corresponding task has the corresponding automatic trigger configuration, and you can also check the corresponding task records and logs to check for any abnormalities.
  2. In TrueWatch, check whether asset information exists in 「Infrastructure / Custom」.
  3. In TrueWatch, check whether there is corresponding monitoring data in 「Metrics」.

Metrics

After configuring Amazon Cloud Monitoring, the default measurement is as follows. You can collect more metrics by configuring Amazon Cloud Monitoring Metrics Details

DEFAULT Level Monitoring

The metrics described in the table below are available at the DEFAULT monitoring level. These metrics are free.

Metrics Available at DEFAULT Monitoring Level
Name When Visible Dimensions Description
ActiveControllerCount After the cluster enters the ACTIVE state. Cluster Name At any given time, only one controller can be active per cluster.
BurstBalance After the cluster enters the ACTIVE state. Cluster Name, Broker ID The remaining balance of I/O burst credits for the EBS volumes in the cluster. Use it to investigate latency or reduced throughput. BurstBalance is not reported for EBS volumes when the baseline performance of the volume is higher than the maximum burst performance. For more information, see I/O Credits and Burst Performance.
BytesInPerSec After creating a topic. Cluster Name, Broker ID, Topic The number of bytes received from clients per second. This metric is available per broker and also per topic.
BytesOutPerSec After creating a topic. Cluster Name, Broker ID, Topic The number of bytes sent to clients per second. This metric is available per broker and also per topic.
ClientConnectionCount After the cluster enters the ACTIVE state. Cluster Name, Broker ID, Client Authentication The number of authenticated active client connections.
ConnectionCount After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of authenticated active connections, unauthenticated connections, and inter-broker connections.
CPUCreditBalance After the cluster enters the ACTIVE state. Cluster Name, Broker ID This metric helps you monitor the CPU credit balance of the broker. If your CPU usage consistently exceeds the baseline utilization of 20%, you might deplete the CPU credit balance, which can negatively affect cluster performance. You can take steps to reduce CPU load. For example, you can reduce the number of client requests or update the broker type to M5.
CpuIdle After the cluster enters the ACTIVE state. Cluster Name, Broker ID The percentage of CPU idle time.
CpuIoWait After the cluster enters the ACTIVE state. Cluster Name, Broker ID The percentage of CPU idle time during pending disk operations.
CpuSystem After the cluster enters the ACTIVE state. Cluster Name, Broker ID The percentage of CPU in kernel space.
CpuUser After the cluster enters the ACTIVE state. Cluster Name, Broker ID The percentage of CPU in user space.
GlobalPartitionCount After the cluster enters the ACTIVE state. Cluster Name The number of partitions in all topics in the cluster (excluding replicas). Since GlobalPartitionCount does not include replicas, the sum of PartitionCount values can be higher than the GlobalPartitionCount value when the replication factor of the topic is greater than 1.
GlobalTopicCount After the cluster enters the ACTIVE state. Cluster Name The total number of topics across all brokers in the cluster.
EstimatedMaxTimeLag After a consumer group consumes a topic. Consumer Group, Topic The estimated time (in seconds) to deplete MaxOffsetLag.
KafkaAppLogsDiskUsed After the cluster enters the ACTIVE state. Cluster Name, Broker ID The percentage of disk space used for application logs.
KafkaDataLogsDiskUsed（Cluster Name, Broker ID Dimensions） After the cluster enters the ACTIVE state. Cluster Name, Broker ID The percentage of disk space used for data logs.
KafkaDataLogsDiskUsed（Cluster Name Dimensions） After the cluster enters the ACTIVE state. Cluster Name The percentage of disk space used for data logs.
LeaderCount After the cluster enters the ACTIVE state. Cluster Name, Broker ID The total number of partition leaders per broker, excluding replicas.
MaxOffsetLag After a consumer group consumes a topic. Consumer Group, Topic The maximum offset lag across all partitions in the topic.
MemoryBuffered After the cluster enters the ACTIVE state. Cluster Name, Broker ID The size of buffered memory (in bytes) for the broker.
MemoryCached After the cluster enters the ACTIVE state. Cluster Name, Broker ID The size of cached memory (in bytes) for the broker.
MemoryFree After the cluster enters the ACTIVE state. Cluster Name, Broker ID The size of free memory (in bytes) available to the broker.
HeapMemoryAfterGC After the cluster enters the ACTIVE state. Cluster Name, Broker ID The percentage of heap memory used after garbage collection.
MemoryUsed After the cluster enters the ACTIVE state. Cluster Name, Broker ID The size of memory (in bytes) used by the broker.
MessagesInPerSec After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of incoming messages per second to the broker.
NetworkRxDropped After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of dropped received packets.
NetworkRxErrors After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of network receive errors for the broker.
NetworkRxPackets After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of packets received by the broker.
NetworkTxDropped After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of dropped transmitted packets.
NetworkTxErrors After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of network transmit errors for the broker.
NetworkTxPackets After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of packets transmitted by the broker.
OfflinePartitionsCount After the cluster enters the ACTIVE state. Cluster Name The total number of partitions in the cluster that are offline.
PartitionCount After the cluster enters the ACTIVE state. Cluster Name, Broker ID The total number of topic partitions per broker, including replicas.
ProduceTotalTimeMsMean After the cluster enters the ACTIVE state. Cluster Name, Broker ID The average produce time (in milliseconds).
RequestBytesMean After the cluster enters the ACTIVE state. Cluster Name, Broker ID The average number of request bytes for the broker.
RequestTime After applying request throttling. Cluster Name, Broker ID The average time (in milliseconds) spent by the broker's network and I/O threads processing requests.
RootDiskUsed After the cluster enters the ACTIVE state. Cluster Name, Broker ID The percentage of the root disk used by the broker.
SumOffsetLag After a consumer group consumes a topic. Consumer Group, Topic The aggregate offset lag across all partitions in the topic.
SwapFree After the cluster enters the ACTIVE state. Cluster Name, Broker ID The size of swap memory (in bytes) available to the broker.
SwapUsed After the cluster enters the ACTIVE state. Cluster Name, Broker ID The size of swap memory (in bytes) used by the broker.
TrafficShaping After the cluster enters the ACTIVE state. Cluster Name, Broker ID An advanced metric indicating the number of packets shaped (dropped or queued) due to exceeding network allocation. PER_BROKER metrics provide more detailed information.
UnderMinIsrPartitionCount After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of partitions on the broker that are not fully managed.
UnderReplicatedPartitions After the cluster enters the ACTIVE state. Cluster Name, Broker ID The number of partitions on the broker that are not fully replicated.
ZooKeeperRequestLatencyMsMean After the cluster enters the ACTIVE state. Cluster Name, Broker ID The average latency (in milliseconds) of Apache ZooKeeper requests from the broker.
ZooKeeperSessionState After the cluster enters the ACTIVE state. Cluster Name, Broker ID The connection state of the broker's ZooKeeper session can be one of the following: NOT_CONNECTED: '0.0', ASSOCIATED: '0.1', CONNECTING: '0.5', CONNECTEDREADONLY: '0.8', CONNECTED: '1.0', CLOSED: '5.0', AUTH_FAILED: '10.0'.

PER_BROKER Level Monitoring

When you set the monitoring level to PER_BROKER, in addition to all DEFAULT level metrics, you will also get the metrics described in the table below. You are charged for the metrics in the table below, while the DEFAULT level metrics remain free. The metrics in this table have the following dimensions: Cluster Name, Broker ID.

Additional Metrics Available at PER_BROKER Monitoring Level
Name When Visible Description
BwInAllowanceExceeded After the cluster enters the ACTIVE state. The number of packets shaped due to inbound aggregate bandwidth exceeding the broker's maximum bandwidth.
BwOutAllowanceExceeded After the cluster enters the ACTIVE state. The number of packets shaped due to outbound aggregate bandwidth exceeding the broker's maximum bandwidth.
ConnTrackAllowanceExceeded After the cluster enters the ACTIVE state. The number of packets shaped due to connection tracking exceeding the broker's maximum. Connection tracking is related to security groups, which track each established connection to ensure that return packets are delivered as expected.
ConnectionCloseRate After the cluster enters the ACTIVE state. The number of connections closed per second per listener. This number is aggregated per listener and then filtered for client listeners.
ConnectionCreationRate After the cluster enters the ACTIVE state. The number of new connections established per second per listener. This number is aggregated per listener and then filtered for client listeners.
CpuCreditUsage After the cluster enters the ACTIVE state. This metric helps you monitor the CPU credit usage on the instance. If your CPU usage consistently exceeds the baseline level of 20%, you might deplete the CPU credit balance, which can negatively affect cluster performance. You can monitor this metric and set alarms to take corrective actions.
FetchConsumerLocalTimeMsMean After providing producers/consumers. The average time (in milliseconds) spent processing consumer requests at the leader.
FetchConsumerRequestQueueTimeMsMean After providing producers/consumers. The average time (in milliseconds) consumer requests waited in the request queue.
FetchConsumerResponseQueueTimeMsMean After providing producers/consumers. The average time (in milliseconds) consumer requests waited in the response queue.
FetchConsumerResponseSendTimeMsMean After providing producers/consumers. The average time (in milliseconds) spent sending consumer responses.
FetchConsumerTotalTimeMsMean After providing producers/consumers. The total average time (in milliseconds) spent by consumers fetching data from the broker.
FetchFollowerLocalTimeMsMean After providing producers/consumers. The average time (in milliseconds) spent processing follower requests at the leader.
FetchFollowerRequestQueueTimeMsMean After providing producers/consumers. The average time (in milliseconds) follower requests waited in the request queue.
FetchFollowerResponseQueueTimeMsMean After providing producers/consumers. The average time (in milliseconds) follower requests waited in the response queue.
FetchFollowerResponseSendTimeMsMean After providing producers/consumers. The average time (in milliseconds) spent sending follower responses.
FetchFollowerTotalTimeMsMean After providing producers/consumers. The total average time (in milliseconds) spent by followers fetching data from the broker.
FetchMessageConversionsPerSec After creating a topic. The number of message conversions per second fetched by the broker.
FetchThrottleByteRate After applying bandwidth throttling. The number of throttled bytes per second.
FetchThrottleQueueSize After applying bandwidth throttling. The number of messages in the throttle queue.
FetchThrottleTime After applying bandwidth throttling. The average fetch throttle time (in milliseconds).
NetworkProcessorAvgIdlePercent After the cluster enters the ACTIVE state. The average percentage of time the network processor was idle.
PpsAllowanceExceeded After the cluster enters the ACTIVE state. The number of packets shaped due to bidirectional PPS exceeding the broker's maximum.
ProduceLocalTimeMsMean After the cluster enters the ACTIVE state. The average time (in milliseconds) spent processing requests at the leader.
ProduceMessageConversionsPerSec After creating a topic. The number of message conversions per second produced by the broker.
ProduceMessageConversionsTimeMsMean After the cluster enters the ACTIVE state. The average time (in milliseconds) spent converting message formats.
ProduceRequestQueueTimeMsMean After the cluster enters the ACTIVE state. The average time (in milliseconds) request messages spent in the queue.
ProduceResponseQueueTimeMsMean After the cluster enters the ACTIVE state. The average time (in milliseconds) response messages spent in the queue.
ProduceResponseSendTimeMsMean After the cluster enters the ACTIVE state. The average time (in milliseconds) spent sending response messages.
ProduceThrottleByteRate After applying bandwidth throttling. The number of throttled bytes per second.
ProduceThrottleQueueSize After applying bandwidth throttling. The number of messages in the throttle queue.
ProduceThrottleTime After applying bandwidth throttling. The average produce throttle time (in milliseconds).
ProduceTotalTimeMsMean After the cluster enters the ACTIVE state. The average produce time (in milliseconds).
RemoteBytesInPerSec After having producers/consumers. The total number of bytes transferred from tiered storage in response to consumer fetches. This metric includes all topic partitions that affect downstream data transfer traffic. Category: Traffic and Error Rate. This is a KIP-405 metric.
RemoteBytesOutPerSec After having producers/consumers. The total number of bytes transferred to tiered storage, including data from log segments, indexes, and other auxiliary files. This metric includes all topic partitions that affect upstream data transfer traffic. Category: Traffic and Error Rate. This is a KIP-405 metric.
RemoteLogManagerTasksAvgIdlePercent After the cluster enters the ACTIVE state. The average percentage of time the remote log manager was idle. The remote log manager transfers data from the broker to tiered storage. Category: Internal Activity. This is a KIP-405 metric.
RemoteLogReaderAvgIdlePercent After the cluster enters the ACTIVE state. The average percentage of time the remote log reader was idle. The remote log reader transfers data from remote storage to the broker in response to consumer fetches. Category: Internal Activity. This is a KIP-405 metric.
RemoteLogReaderTaskQueueSize After the cluster enters the ACTIVE state. The number of tasks responsible for reading from tiered storage and waiting to be scheduled. Category: Internal Activity. This is a KIP-405 metric.
RemoteReadErrorPerSec After the cluster enters the ACTIVE state. The total error rate of read requests sent by the specified broker to tiered storage to retrieve data in response to consumer fetches. This metric includes all topic partitions that affect downstream data transfer traffic. Category: Traffic and Error Rate. This is a KIP-405 metric.
RemoteReadRequestsPerSec After the cluster enters the ACTIVE state. The total number of read requests sent by the specified broker to tiered storage to retrieve data in response to consumer fetches. This metric includes all topic partitions that affect downstream data transfer traffic. Category: Traffic and Error Rate. This is a KIP-405 metric.
RemoteWriteErrorPerSec After the cluster enters the ACTIVE state. The total error rate of write requests sent by the specified broker to tiered storage to transfer data upstream. This metric includes all topic partitions that affect upstream data transfer traffic. Category: Traffic and Error Rate. This is a KIP-405 metric.
ReplicationBytesInPerSec After creating a topic. The number of bytes received per second from other brokers.
ReplicationBytesOutPerSec After creating a topic. The number of bytes sent per second to other brokers.
RequestExemptFromThrottleTime After applying request throttling. The average time (in milliseconds) spent by the broker's network and I/O threads processing requests exempt from throttling.
RequestHandlerAvgIdlePercent After the cluster enters the ACTIVE state. The average percentage of time the request handler threads were idle.
RequestThrottleQueueSize After applying request throttling. The number of messages in the throttle queue.
RequestThrottleTime After applying request throttling. The average request throttle time (in milliseconds).
TcpConnections After the cluster enters the ACTIVE state. Displays the number of incoming and outgoing TCP segments with the SYN flag set.
TotalTierBytesLag After creating a topic. The total number of bytes of data that are eligible for tiering on the broker but have not yet been transferred to tiered storage. These metrics show the efficiency of upstream data transfer. As latency increases, the amount of data not present in tiered storage also increases. Category: Archive Latency. This is not a KIP-405 metric.
TrafficBytes After the cluster enters the ACTIVE state. Displays the network traffic between clients (producers and consumers) and brokers in total bytes. Traffic between brokers is not reported.
VolumeQueueLength After the cluster enters the ACTIVE state. The number of read and write operation requests waiting to be completed during the specified time period.
VolumeReadBytes After the cluster enters the ACTIVE state. The number of bytes read during the specified time period.
VolumeReadOps After the cluster enters the ACTIVE state. The number of read operations during the specified time period.
VolumeTotalReadTime After the cluster enters the ACTIVE state. The total number of seconds spent completing all read operations during the specified time period.
VolumeTotalWriteTime After the cluster enters the ACTIVE state. The total number of seconds spent completing all write operations during the specified time period.
VolumeWriteBytes After the cluster enters the ACTIVE state. The number of bytes written during the specified time period.
VolumeWriteOps After the cluster enters the ACTIVE state. The number of write operations during the specified time period.

PER_TOPIC_PER_BROKER Level Monitoring

When you set the monitoring level to PER_TOPIC_PER_BROKER, in addition to all PER_BROKER and DEFAULT level metrics, you will also get the metrics described in the table below. Only DEFAULT level metrics are free. The metrics in this table have the following dimensions: Cluster Name, Broker ID, Topic.

Important: For Amazon MSK clusters using Apache Kafka 2.4.1 or later, the metrics in the table below only appear after their value first becomes non-zero. For example, to see BytesInPerSec, one or more producers must first send data to the cluster.

Additional Metrics Available at PER_TOPIC_PER_BROKER Monitoring Level
Name When Visible Description
FetchMessageConversionsPerSec After creating a topic. The number of fetched message conversions per second.
MessagesInPerSec After creating a topic. The number of messages received per second.
ProduceMessageConversionsPerSec After creating a topic. The number of produced message conversions per second.
RemoteBytesInPerSec After you create a topic and the topic is producing/consuming. The number of bytes transferred from tiered storage in response to consumer fetches for the specified topic and broker. This metric includes all partitions in the topic that affect downstream data transfer traffic for the specified broker. Category: Traffic and Error Rate. This is a KIP-405 metric.
RemoteBytesOutPerSec After you create a topic and the topic is producing/consuming. The number of bytes transferred to tiered storage for the specified topic and broker. This metric includes all partitions in the topic that affect upstream data transfer traffic for the specified broker. Category: Traffic and Error Rate. This is a KIP-405 metric.
RemoteReadErrorPerSec After you create a topic and the topic is producing/consuming. The error rate of read requests sent by the specified broker to tiered storage to retrieve data in response to consumer fetches for the specified topic. This metric includes all partitions in the topic that affect downstream data transfer traffic for the specified broker. Category: Traffic and Error Rate. This is a KIP-405 metric.
RemoteReadRequestsPerSec After you create a topic and the topic is producing/consuming. The number of read requests sent by the specified broker to tiered storage to retrieve data in response to consumer fetches for the specified topic. This metric includes all partitions in the topic that affect downstream data transfer traffic for the specified broker. Category: Traffic and Error Rate. This is a KIP-405 metric.
RemoteWriteErrorPerSec After you create a topic and the topic is producing/consuming. The error rate of write requests sent by the specified broker to tiered storage to transfer data upstream for the specified topic. This metric includes all partitions in the topic that affect upstream data transfer traffic for the specified broker. Category: Traffic and Error Rate. This is a KIP-405 metric.

PER_TOPIC_PER_PARTITION Level Monitoring

When you set the monitoring level to PER_TOPIC_PER_PARTITION, in addition to all metrics from PER_TOPIC_PER_BROKER, PER_BROKER, and DEFAULT levels, you will also get the metrics described in the table below. Only DEFAULT level metrics are free. The metrics in this table have the following dimensions: Consumer Group, Topic, Partition.

Additional Metrics Available at PER_TOPIC_PER_PARTITION Monitoring Level
Name When Visible Description
EstimatedTimeLag After a consumer group consumes a topic. The estimated time (in seconds) to deplete the partition offset lag.
OffsetLag After a consumer group consumes a topic. The lag of the partition-level consumer in terms of offset.

Objects

AWS MSK object data is currently unavailable.