Changelog (2022)¶

This document records the update content descriptions for each release of TrueWatch, including DataKit, TrueWatch Best Practices, TrueWatch Integration Documentation, and TrueWatch.

December 29, 2022¶

TrueWatch Updates¶

New MFA Authentication Management¶

TrueWatch provides MFA authentication management to add an extra layer of security beyond account username and password. After enabling MFA authentication, you will need to perform secondary verification during login, helping to improve your account security. For more details, refer to the document MFA Management.

TrueWatch supports configuring IP whitelists for workspaces to restrict incoming users. After enabling the IP whitelist, only requests from IP sources in the whitelist can log in normally, while requests from other sources will be denied access. For more details, refer to the document IP Whitelist.

Note: The IP whitelist can only be set by administrators and owners. Additionally, 'Owners' are not restricted by the IP whitelist access limitations.

New Service Network Topology¶

In a K8S environment, you can use the Service network topology diagram to view request relationships, response times, error rates, etc., between various Services in the K8S environment. When a connection issue is found with a Service, you can click to view the logs corresponding to that Service to locate the problem. For more details, refer to the document Service Network.

Other Feature Optimizations¶

Pipeline usage optimization
Log clustering analysis supports custom fields
Oversized logs after splitting support viewing their associated logs
Backup log display optimization
Data authorization sensitive fields support all data types except Metrics

DataKit Updates¶

Prometheus collector supports data collection via Unix Socket
Allows non-root users to run DataKit
Optimized eBPF httpflow protocol determination
Optimized Datakit installation and upgrade commands on Windows
Optimized Pythond usage encapsulation
Pipeline provides more detailed operation error information
Pipeline reftable provides a SQLite-based localized storage implementation

For more DataKit updates, refer to DataKit Version History.

Intelligent Inspection Updates¶

Official Intelligent Inspections Discontinued¶

The official intelligent inspections 'Memory Leak', 'Application Performance Detection', and 'Disk Usage Rate' are discontinued from today. Users can configure them via the Self-built Inspection script library.

New Cloud Account Instance Dimension Bill Inspection¶

Cloud account instance dimension bill inspection helps users manage abnormal cost alerts at the cloud service instance level, predict cost situations, and provides users with prompts for high-growth, high-consumption instances and bill visualization capabilities, supporting multi-dimensional visualization of cloud service resource consumption.

New Alibaba Cloud Preemptible Instance Survival Inspection¶

Since the market price of preemptible instances fluctuates with supply and demand, a bidding mode needs to be specified when creating a preemptible instance. A preemptible instance can only be successfully created when the real-time market price of the specified instance specification is lower than the bid and inventory is sufficient. Therefore, inspection of preemptible instances for cloud assets is particularly important. Through inspection, when a preemptible instance is about to be released, it will prompt the latest prices for all availability zones of the current specification's preemptible instances and the historical price of that preemptible instance, and provide appropriate handling suggestions.

For more intelligent inspection updates, refer to Intelligent Inspection Changelog.

December 15, 2022¶

TrueWatch Updates¶

New Profile and Network Billing Items¶

TrueWatch adds Profile and Network billing items. Profile counts the number of application performance Profiles reported within the workspace on the current day. Network counts the number of hosts (host) within the workspace that reported network data on the current day. For more details, refer to the document Billing Methods.

TrueWatch Registration Process Optimization¶

TrueWatch has simplified the registration process to help you quickly complete registration and start using TrueWatch. For more details, refer to the document Register TrueWatch Commercial Plan.

New Warroom Feature¶

TrueWatch provides the 'Warroom' dashboard. You can get an overview of key metric trends across all workspaces in the 'Warroom' dashboard, helping you quickly understand the key data situation of each workspace and promptly discover and resolve abnormal issues. For more details, refer to the document Key Metrics Management.

New Dashboard Carousel Function¶

TrueWatch provides a carousel function for multiple associated business dashboards. After configuration, you can display them on large screens. For more details, refer to the document Carousel Dashboard.

Workspace Binding External SLS Index¶

TrueWatch adds support for binding SLS indexes, helping you to uniformly and quickly view and analyze your log data. Additionally, TrueWatch adds support for field mapping functionality, helping you map SLS index fields for clustering analysis. For more details, refer to the document Log Index.

Trace Explorer Chart Display Optimization¶

Added 'Span Count' and 'Error Span Count' trend charts in the trace explorer. You can quickly view metric trends for Span count, Error Span count, request count, Error request count, and response time through the explorer's charts.

Other Feature Optimizations¶

View variable interaction optimization
Binding built-in view configuration interaction optimization
Workspace overdue lock process optimization
Workspace invitation user experience optimization
Help documentation adds an evaluation system, including 'document good' and 'document poor'
Monitor batch function optimization
SLO adds enable/disable functionality
Fuzzy matching Wildcard left * matching query limit optimization

DataKit Updates¶

Added Golang Profiling integration
logfwd supports injecting the image field via LOGFWD_TARGET_CONTAINER_IMAGE
trace collector:
- Optimized error-stack/error-message format issues
- SkyWalking compatibility adjustments, supporting full 8.X series
eBPF httpflow adds pid/process_name field, optimized kernel version support
datakit.yaml has been adjusted, recommend updating yaml
GPU graphics card collection supports remote mode

For more DataKit updates, refer to DataKit Version History.

Intelligent Inspection Updates¶

Intelligent Inspection Adds Guide Page¶

TrueWatch natively supports three intelligent inspection templates: Memory Leak, Disk Usage Rate, and Application Performance Detection. After this update, existing workspaces support manually deleting the default intelligent inspection templates, with a prompt for the system-wide discontinuation time. Newly created workspaces no longer provide these three intelligent inspections by default, adding a new user guide prompt.

Intelligent Inspection Components Support Configuring Custom Redirect Links¶

Supports configuring custom redirect links when setting up self-built inspections. For how to configure self-built inspections, refer to the document Self-built Inspection.

For more intelligent inspection updates, refer to Intelligent Inspection Changelog.

Best Practices Updates¶

Insight
- Scene - SpringBoot Project External Tomcat Scenario Link Observability.

For more best practices updates, refer to Best Practices Version History.

December 1, 2022¶

TrueWatch Updates¶

APM Adds Service List Feature¶

APM Service List supports real-time viewing of ownership, dependencies, performance, associated dashboards, and correlation analysis for different services, helping to quickly discover and resolve service performance issues, assisting teams in efficiently building and managing large-scale end-to-end distributed applications. For more details, refer to the document Service List.

Dashboard Chart Usage Experience Optimization¶

Added manual input for chart units, colors, and aliases. You can customize preset units, colors, and aliases according to current specifications, e.g., unit input format: aggregation function(metric), like last(usage_idle)
Time series chart adds Y-axis configuration, allowing manual configuration of Y-axis maximum and minimum values
Time series chart, pie chart add grouping display; when enabled, the legend only displays label values
Heatmap, China map, world map add gradient intervals, including automatic and custom
Bar chart, histogram display optimization

For more details, refer to the document Visualization Charts.

View Variable Usage Experience Optimization¶

View variables now support multi-selection
View variable configuration adds option to set multiple default values

For more details, refer to the document View Variables.

RUM Session Explorer Adjustment¶

In the RUM Session explorer, added 'Session' and 'All Records' view lists: - Session: Statistics of deduplicated Session data within the current time range, displaying the latest Session data by default. - All Records: Displays all reported Session data; the same Session may correspond to multiple data records.

Note: In the Session list, adding display columns is supported. If a user adds a display column for a field that does not exist in the current Session data, there may be empty data in the session list. For more details, refer to the document Session.

Events Add Mobile App Redirect Option¶

After users receive an alert event notification, clicking 'Go to TrueWatch to view' can redirect to the TrueWatch mobile app to view detailed information about the abnormal event.

Other Feature Optimizations¶

New user guide page optimization, supports minimization
RUM application list optimization, adds time widget custom interval switching query
Host adding Label interaction optimization

DataKit Updates¶

Added Python Profiling integration
Pythond adds custom event reporting functionality
netstat supports metric collection for specific ports
Optimized Redis collector CPU usage rate collection, added new metric fields
Optimized logfwd collector configuration
Completed host object field collection, added network, disk, and other related fields

For more DataKit updates, refer to DataKit Version History.

Intelligent Inspection Updates¶

Kubernetes Pod Abnormal Restart Inspection
MySQL Performance Inspection
Server Application Error Inspection
Memory Leak Inspection
Disk Usage Rate Inspection
Application Performance Inspection
Frontend Application Log Error Inspection

For more intelligent inspection updates, refer to Intelligent Inspection Changelog.

Best Practices Updates¶

Monitoring
- Application Performance Monitoring (APM) - Performance Optimization - Using async-profiler for Application Performance Tuning

For more best practices updates, refer to Best Practices Version History.

November 17, 2022¶

TrueWatch Updates¶

Pipeline Usage Experience Optimization¶

Pipeline supports multi-selection for filter condition configuration
Supports setting any Pipeline script as the 'Default Pipeline Script'. If the current data type does not match other Pipeline scripts during Pipeline processing matching, data will be processed according to the rules of the default Pipeline script.
Script function categorization

Event Optimization¶

Supports writing user custom events. For more details, refer to the document Write Event Data via openAPI.
Unrecovered event explorer left side adds quick filters

Custom Explorer Supports Selecting More Data Types¶

When creating an explorer in a scene, supports selecting 5 data types: Log, APM, RUM, Security Check, Profile. Cannot be changed after saving.

Member Management Adds Member Grouping Function¶

Member management adds a member grouping function. You can edit member groups, and a member can belong to multiple groups. Also, when configuring notification targets for alert strategies, supports configuring member group tags.

Optimized Built-in View Binding Function¶

In the scene 'Built-in Views' - 'User Views', binding fields now supports fuzzy matching. After fields are bound, corresponding built-in views can be associated and viewed in the explorer.

New One-Click Import/Export of Dashboards, Custom Explorers, Monitors within Workspace¶

In 'Management' - 'Basic Settings', added one-click import/export of JSON configurations for dashboards, custom explorers, monitors, etc., within the workspace. You can import dashboards, custom explorers, monitors, etc., from other workspaces into the current workspace.

APM Log Page Adds Custom Association Fields¶

On the APM detail page, in the associated logs section, if you have administrator or higher permissions, you can customize association fields. Click the settings button next to the association field, select the fields to associate in the pop-up dialog, supports manual input and drag-and-drop ordering, confirm to complete configuration.

Other Feature Optimizations¶

When filter conditions use fuzzy match and fuzzy not match, supports left * matching
Infrastructure analysis dimension optimization, host last reported time on host detail page
Supports user quick login to the last single sign-on
APM blacklist adds support for filtering 'All Services'
Users registered at the 'China Region 4 (Guangzhou)' site optimized process for upgrading to Commercial Plan
Field descriptions support viewing in quick filters, display columns, etc.

DataKit Updates¶

Added SNMP Collector
Added IPMI Collector
Added batch injection DDTrace-Java tool
Latest DDTrace-Java SDK enhances SQL obfuscation functionality
Remote Pipeline optimization
- Pipeline supports source mapping relationship configuration, facilitating batch configuration between Pipeline and data sources
- Pipeline provides function classification information, facilitating remote Pipeline writing
Optimized Kafka Message Subscription, no longer limited to obtaining SkyWalking related data, also supports rate limiting, multi-version coverage, sampling, and load balancing settings
By providing additional configuration parameters (ENV_INPUT_CONTAINER_LOGGING_SEARCH_INTERVAL), alleviates short-lived Pod log collection issues
In pure container environments, supports configuring container log collection via label method
Added Pipeline functions
- sample(): Sampling function
- b64enc(): Base64 encoding function
- b64dec(): Base64 decoding function
- append(): List append function

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Insight
- TrueWatch Tips (Skills) - DataKit Configures HTTPS
Monitoring
- Application Performance Monitoring (APM) - ddtrace common parameter usage
- Application Performance Monitoring (APM) - ddtrace-api usage guide
- Application Performance Monitoring (APM) - ddtrace sampling
- Application Performance Monitoring (APM) - ddtrace log correlation

For more best practices updates, refer to Best Practices Version History.

November 11, 2022¶

TrueWatch Adds 'China Region 4 (Guangzhou)' Site¶

TrueWatch supports multi-site registration and login, adding the 'China Region 4 (Guangzhou)' site. Combined with the existing 'China Region 1 (Hangzhou)', 'China Region 2 (Ningxia)', and 'International Region 1 (Oregon)', it comprehensively covers different regions domestically and internationally, providing users with comprehensive data observability.

Note that accounts and data for different TrueWatch sites are independent and cannot be shared or migrated between sites. You can choose a suitable site for registration and login based on your resource usage. Currently, TrueWatch supports the following four sites.

Site	Login URL	Operator	Settlement Method
China Region 1 (Hangzhou)	https://auth.truewatch.com/	Alibaba Cloud	TrueWatch Enterprise Account, Cloud Account
China Region 2 (Ningxia)	https://aws-auth.truewatch.com/	AWS	TrueWatch Enterprise Account, Cloud Account
International Region 1 (Oregon)	https://us1-auth.truewatch.com/	AWS	TrueWatch Enterprise Account, Cloud Account
China Region 4 (Guangzhou)	https://cn4-auth.truewatch.com/	Huawei Cloud	TrueWatch Enterprise Account, Huawei Cloud Prepaid Card

November 3, 2022¶

TrueWatch Updates¶

New TrueWatch and SLS Joint Solution¶

TrueWatch adds an SLS storage solution, supporting Alibaba Cloud SLS users to quickly use TrueWatch for data viewing and analysis. When registering/upgrading to the Commercial Plan in TrueWatch, after selecting 'Alibaba Cloud Account Settlement', you can choose the SLS storage solution to store data under your own Alibaba Cloud account, achieving data exclusivity. For more details, refer to the document SLS Storage.

The joint solution of TrueWatch and SLS supports you to synchronize data from your Alibaba Cloud account to the TrueWatch workspace for query and analysis through the bind log index function. You can click 'Bind Index' in the workspace 'Log' - 'Index' to synchronize log data from the index. For more details, refer to the document Log Index.

Optimized New User Guide Page¶

TrueWatch adds new user guide pages for major functional modules: Scene, Event, Infrastructure, Metrics, Log, APM, RUM, Synthetic Tests, Security Check, CI Visualization, Monitoring. Also optimized the workspace welcome homepage, making it easier for you to understand and use TrueWatch through the new user guides.

Added 3 Intelligent Inspection Configuration Documents¶

Cloud Account Bill Inspection: Helps users manage cloud service budget alerts, abnormal cost alerts, predict cost situations, and provides users with visualization capabilities, supporting multi-dimensional visualization of cloud service resource consumption.
Frontend Application Log Error Inspection: Helps discover new error messages (clustered Error Message) that appeared in frontend applications in the past hour, helping development and operations promptly fix code to avoid continuous damage to customer experience over time.
Alibaba Cloud Asset Inspection: Helps users gain more understanding of cloud vendor product performance status.

New Trace Error Tracking Explorer¶

In APM, added an error tracking explorer, helping you quickly view similar errors in traces and their distribution, quickly locating performance issues. For more details, refer to the document Trace.

Optimized Time Series Chart, Overview Chart Period-over-Period Comparison Function¶

Period-over-period comparison function no longer interacts with locked time. The button is displayed by default and is in the off state. When period-over-period comparison is enabled, comparison dimensions support 4 options: Hour (compared with one hour ago), Day (compared with one day ago), Week (compared with one week ago), Month (compared with one month ago). For more details, refer to the document Period-over-Period Comparison.

Other Feature Optimizations¶

Dashboard/Note/Explorer adds a 'Save Snapshot' button in 'Settings'.
Time widget adds more options.
Pipeline and blacklist functions add import, batch export, batch delete functions.
Intelligent inspection adds dashboard, bar chart chart components.
Func page optimization under the Integration menu.

DataKit Updates¶

Improved Prometheus ecosystem compatibility, added ServiceMonitor and PodMonitor collection recognition
Added Java Profiling integration based on async-profiler
eBPF collection adds interval parameter to facilitate adjusting the amount of data collected
All remote collectors default to using their collection address as the value for the host field, avoiding possible misunderstandings of the host field value during remote collection
APM data collected by DDTrace can automatically extract error-related fields, facilitating better APM error tracking at the center
MySQL collector adds additional fields Com_commit/Com_rollback collection

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Monitoring
- Application Performance Monitoring (APM) - Kafka Observability Best Practices
Cloud Platform Integration
- Alibaba Cloud - Alibaba Cloud ACK Integration with TrueWatch

For more best practices updates, refer to Best Practices Version History.

October 20, 2022¶

TrueWatch Updates¶

Monitoring Optimization¶

Added Outlier Detection¶

Outlier detection is an algorithm that detects whether metrics/statistical data of detection objects under specific groupings have outlier deviations. If inconsistencies exceed a certain degree, outlier detection abnormal events are generated for subsequent alert tracking. For more details, refer to the document Outlier Detection.

Mutation Detection Logic Optimization¶

The new version of mutation detection determines whether abnormal situations occur by comparing the absolute or relative (%) change value of the same metric between two different time periods. Mostly used for tracking peaks or data changes of a metric, generating events more accurately when abnormalities occur for record-keeping. For more details, refer to the document Mutation Detection.

Range Detection Logic Optimization¶

The new version of range detection performs anomaly detection on metric data within the selected detection range time period. When the proportion of mutated abnormal data points exceeds the set percentage, range detection abnormal events are generated. Mostly used for monitoring data/metrics with stable trends. For more details, refer to the document Range Detection.

Other Optimizations¶

Adjusted read-only member permissions, supporting viewing configuration content of monitors, intelligent inspections, SLO, mute management, and alert strategies
Data gap configuration selection 'Trigger data gap event' and 'Trigger recovery event' adjusted to time range configuration, supports manual input

Supports configuring monitor detection based on 'Event' data

Alert configuration adds 'Info' event notification level selection

Scene Chart Optimization¶

Added Histogram Chart Component¶

Histogram, also known as mass distribution chart, is used to represent data distribution, a common statistical chart. Generally, the horizontal axis represents data intervals, and the vertical axis represents distribution. For more details, refer to the document Histogram.

Chart Function Optimization¶

Overview chart, treemap, funnel chart add time slicing function
Time series chart 'View similar trend metrics' adjusted from supporting only metric queries to supporting all data types, including Log, APM, RUM, etc.
Top List supports viewing all content beyond chart width
Heatmap display optimization

Explorer Optimization¶

Explorer Supports Analysis Mode¶

Added 'Analysis' function in Log, APM, RUM, Synthetic Testing, Security Check, CI explorers. Infrastructure adjusted 'Grouping' to 'Analysis', supporting multi-dimensional analysis statistics based on 1-3 tags to reflect data distribution characteristics and trends across different dimensions over time. In analysis mode, supports multiple data chart analyses, including time series chart, top list, pie chart, and treemap. For more details, refer to the document Explorer Analysis Mode.

Optimized Associated Log Viewing Experience¶

In Infrastructure, APM, RUM, CI Visualization explorer detail pages, optimized associated log viewing experience. By default, log content is displayed according to the 'Maximum Display Rows' and 'Display Columns' set in the log explorer, supports custom adjustment via 'Display Columns'.

Log Index Optimization¶

After setting an index in 'Log' - 'Index', log-related queries add an index option, defaulting to index default. You can query and analyze log data based on the index you set, involving functions including log explorer, scene custom explorer, chart log query, log monitor, etc. For more details, refer to the document Log Index.

Optimized Metric Analysis Chart Query¶

Added time interval and legend options in metric analysis, adjusted legend aggregation function display order. For more details, refer to Metric Analysis.

Other Feature Optimizations¶

In the RUM application list, after clicking into an application, a dropdown menu option is added in the upper left corner to help users quickly switch and view different application data
Added Pipelines shortcut entries in the Metrics, RUM, APM, Infrastructure, Security Check directories

DataKit Updates¶

DataKit collector configuration and Pipeline support synchronization via configuration centers like etcd/Consul
Prometheus Remote Write optimization
- Collection supports filtering tags via regex
- Supports filtering measurement names via regex
Pipeline optimization
- Optimized functions like grok(), enabling them to be used in if/else statements to determine if operations take effect
- Added match() function
- Added cidr() function
Process collector adds details field for open file list
Improved disk caching and queue processing for externally integrated data (T/R/L)
Monitor adds overage usage prompt: at the bottom of the monitor, if current space usage exceeds the limit, a red text Beyond Usage prompt appears
Optimized log collection position function; in container environments, the file will be mounted to the host to avoid losing original position records after DataKit restart
Optimized collection latency issues in sparse log scenarios

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Monitoring
- Infrastructure Monitoring (ITIM) - Ansible Batch Processing Practice.
Cloud Native
- Log - TrueWatch Collects Amazon ECS Logs.

For more best practices updates, refer to Best Practices Version History.

September 29, 2022¶

TrueWatch Updates¶

New Field Management¶

TrueWatch supports unified management of field data in the current workspace, including system fields and custom fields. You can view field descriptions in scene chart queries, monitor detection metrics, DQL query simple query mode, etc., helping you quickly understand field meanings and apply fields. The image below shows viewing the host field description in a scene chart query. For more details, refer to the document Field Management.

Optimized Metrics Function¶

Metric Explorer Transformation¶

The original 'Metric Explorer' is renamed to 'Metric Analysis', supporting users to visualize query data in time series chart mode (line chart, area chart, bar chart). For more details, refer to Metric Analysis.

Optimized Metric Management¶

Supports searching for 'Measurement' and 'Metric' in 'Metric Management'.
Supports modifying metric 'Unit' and 'Description' in 'Metric Management'.
Supports viewing tag description information in 'Metric Management'.

For more details, refer to Metric Management.

Added Export Event Content as PDF Format¶

On the event detail page, the Jump to Monitor button adds text description. Click the button, you can choose Export JSON File and Export PDF File to obtain all key data corresponding to the current event.

Monitor Adjustments¶

Monitor name does not support customization; monitor name is generated synchronously after entering the event title.
Monitor supports 'info' event generation logic. Trigger conditions can choose to enable or disable info: normal detection results also generate events.
Monitor configuration step order adjusted to: Detection Configuration - Event Notification - Association.

Note: In the latest version, 'Monitor Name' will be generated synchronously after entering the 'Event Title'. Old monitors may have inconsistencies between 'Monitor Name' and 'Event Title'. For a better user experience, please synchronize to the latest as soon as possible, supporting one-click replacement of the event title.

Mute Rules Support Dynamic Configuration¶

In Mute Management, optimized mute scope, supporting options like monitor, intelligent inspection, self-built inspection, SLO, alert strategy, etc. Added tag options.

Note: Mute scope is required, tags are optional. Mute only takes effect when both 'Mute Scope' and 'Tag' conditions are met simultaneously. For example, if mute scope selects Application Performance Detection and Disk Usage Rate, and tag selects: host:izbp152ke14timzud0du15z, when conditions simultaneously match Application Performance Detection and host:izbp152ke14timzud0du15z OR conditions simultaneously match Disk Usage Rate and host:izbp152ke14timzud0du15z, the alert notification is muted and will not be sent to alert notification targets.

Optimized Pipeline Configuration Page¶

In Text Processing (Pipeline), supports adding multiple sample parsing tests, fixed the issue where one-click sample retrieval for metric data was incorrect. For more details, refer to Text Processing (Pipeline).

Other Feature Optimizations¶

Note adds global locked time configuration; after configuring global locked time, all charts on that note page display data according to that locked time
Unrecovered event query changed to data from the last 48 hours, supports manual refresh
RUM supports same-name user view override logic
RUM Web, WeChat Mini Program, and uniapp-based mini program integration initialization parameters add isIntakeUrl configuration, used to determine whether to collect corresponding resource data based on request resource URL, defaults to collecting all
Intelligent inspection adds support for self-built inspection
Generate metric page operation column adjustment, adds 'Open in Metric Analysis' and 'Open in Metric Management' operation icons
Management navigation menu position adjustment, SSO management moved to Member Management, Notification Targets management moved to Monitoring, Built-in Views moved to Scene

Best Practices Updates¶

Cloud Native: Multiple Kubernetes Cluster Metric Collection Best Practices.

For more best practices updates, refer to Best Practices Version History.

September 15, 2022¶

TrueWatch Updates¶

Added Log Explorer DQL Search Mode¶

In the log explorer, you can click the switch icon next to the search bar to switch to DQL manual input query mode, supporting custom input of filter conditions and search conditions.

Filter conditions: Support any combination of and / or, support using () parentheses to indicate search execution priority, support operators like =, !=, etc.
Search conditions: Support searching logs via the DQL function query_string().

For more details, refer to the document DQL Search.

Optimized APM¶

Trace Explorer Detail Page Optimization¶

Optimized detail page layout display, added duration, http method, http url, http status code, TraceId, etc.
Flame graph adds thumbnail, double-click Span to zoom, click service name to highlight corresponding Span
Span list adds search, click select Span switches to flame graph synchronized display
Service call relationship adds search

For more details, refer to the document Trace Explorer.

Profile Explorer Detail Page Optimization¶

Added quick operations in the Profile explorer detail page, including search, copy, and click select to view methods. For more details, refer to the document Quick Operation Instructions.

Optimized Monitor Event Notification Content Edit Mode¶

When creating a monitor, the event notification content sent when trigger conditions are met supports input and preview of markdown format text information, supports using template variables.

Added Mute Management Support for Configuring Periodic Mutes¶

Mute management adds periodic mute function. In mute time, you can choose 'Only Once' or 'Repeat'. For more details, refer to Mute Management.

Other Feature Optimizations¶

Scene dashboard grouping display optimization, scene note optimization text component Markdown format
Infrastructure explorer display optimization, added display column CPU usage rate, MEM usage rate, etc., prompt information; 'Pods', 'Deployments', 'Services' explorer detail pages add view and copy corresponding YAML file
Metric explorer removes list view mode, retains tiled view mode and mixed view mode
Log multi-index supports jump view
Explorer quick filter value TOP 5 supports viewing proportion count, explorer time field format optimization, default display format is 2022/09/15 20:53:40
Trace explorer time series chart adds legend display, quick filters add HTTP related fields

DataKit Updates¶

Added automatic cloud synchronization function, no longer need to manually specify cloud vendor
Supports synchronizing k8s labels as tags to pod metrics and logs
Supports collecting various k8s yaml information to corresponding object data
Trace collection supports automatic extraction of some key meta information
Supports specifying installation source address during installation to simplify offline installation process
Pipeline new features:
- Added for loop/dictionary/array support
- Added arithmetic expression support
- Pipeline error information will be displayed on collected data
- If time field parsing fails, supports automatic correction of the time field (time) to avoid time display issues on the console page
- Added len() function

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Monitoring - Middleware - Insight MySQL.

For more best practices updates, refer to Best Practices Version History.

September 1, 2022¶

TrueWatch Updates¶

Billing Update¶

TrueWatch billing item backup log adjusted billing rules. After adjustment, backup logs are charged at 0.007 yuan per 1 GB of data/day. You can adjust corresponding data retention policies based on actual needs to save cost. For more details, refer to the document Billing Methods.

Help Documentation Directory Update¶

Split changelog directory, supports viewing TrueWatch update content by year.
Added Custom Development directory, can view documents for custom collector (Pythond), custom security inspection (Scheck), custom DDtrace, etc.
Added Private Deployment directory, can view documents related to TrueWatch deployment.

Time Series Chart Added Event Correlation Analysis¶

Time series chart added event correlation analysis, supports matching abnormal events related to selected fields by 'Adding Filter Fields', achieving the purpose of displaying time series data and events together, helping users perceive whether related events occurred during data fluctuations while viewing trends, to locate issues.

After configuring event correlation in a time series chart, on the time axis, if there are event records related to the selected association field (e.g., 'host:DESKTOP-NEN89U3'), they will be marked with shaded blocks.
After configuring event correlation in a time series chart, in analysis mode, clicking the highlighted shaded block allows viewing abnormal events related to the selected field (e.g., 'host:DESKTOP-NEN89U3').

For more details, refer to the document Time Series Chart.

Log Added Multi-Index Mode¶

TrueWatch supports setting multiple log indexes, filtering logs that meet conditions to be saved in different log indexes, and by selecting different data storage policies for log indexes, helps users save log data storage costs. For more details, refer to the document Log Index.

Optimized Backup Log Rules¶

Backup log new rule creation moved to the Log Index menu. Backup cycle changed from backing up previous day's logs at 0:00 daily to executing rule validation and backup every 5 minutes, i.e., configured backup rules can view backup log data within at most 5 minutes. For more details, refer to the document Backup Log.

Optimized Log Context¶

When viewing context on the log detail page and opening a new log explorer, it will use 'host', 'source', 'service', 'filename' as filter conditions to view data before and after the current log time.

Optimized RUM¶

Added Custom RUM Application ID¶

RUM supports users customizing an application ID as a unique identifier for the current workspace (original TrueWatch application ID is globally unique, i.e., application IDs cannot be repeated across spaces). Different workspaces can use the same application ID as a unique identifier for the same application, used for SDK data upload matching. For more details, refer to documents:

Web Application Integration
Android Application Integration
iOS Application Integration
Mini Program Application Integration

Added RUM Network Request Error Correlation Trace View¶

RUM supports users associating and viewing network (network) error related traces in the RUM explorer.

Intelligent Inspection Comprehensive Upgrade¶

Intelligent inspection list adjusted 'Grouping' to 'Alert Strategy', supports selecting alert strategy for intelligent inspection during editing.
Intelligent inspection related event detail pages comprehensively adjusted, including related pages like basic attributes, event details, root cause analysis, alert notification, history records, associated events, etc. Also optimized abnormal chart analysis, added data association jump links, abnormal interval prompts, root cause analysis interval prompts, legend display operations, etc.

For more details, refer to Intelligent Inspection.

Optimized Monitoring¶

Adjusted Grouping to Alert Strategy¶

In 'Monitoring', adjusted 'Group Management' page to 'Alert Strategy Management', creating alert strategies to filter monitors and set alert notifications.

In Monitor, original 'Grouping' filter adjusted to 'Alert Strategy' filter, supports selecting alert strategy when configuring monitor, supports clicking monitor's alert strategy to modify alert settings.

Optimized Monitor Configuration¶

Adjusted monitor configuration process steps, including detection frequency, detection interval, alert strategy, etc.
Optimized detection metrics, supports configuring outer functions (fx), including 'Label Filtering', 'Transformation Functions', 'Data Gap Filling'; supports switching to DQL query mode.
Optimized trigger condition text prompts.
Optimized event notification, removed some monitor custom data gap event titles and content; adjusted template variable prompt text; adjusted grouping to alert strategy, and supports editing alert strategy.
Optimized log detection, APM metric detection, RUM metric detection 'Detection Metric' configuration.

For more details, refer to Monitor Management.

Added 'Infrastructure Liveness Detection'¶

'Infrastructure Liveness Detection' is used to monitor the operational status of infrastructure. For more details, refer to Infrastructure Liveness Detection.

Added 'Process Anomaly Detection'¶

'Process Anomaly Detection' is used to monitor process data within the workspace, supports setting trigger alerts for one or more field types of process data. For more details, refer to Process Anomaly Detection.

Note: Infrastructure object detection has been removed. Existing related monitors can continue to be used, but new creation is no longer supported.

Optimized 'APM Metric Detection'¶

'APM Metric Detection' adds 'Trace Statistics' detection method, counting the number of traces meeting conditions within a specified time, triggering abnormal events when exceeding custom thresholds. Can be used for service trace abnormal error notifications. For more details, refer to APM Metric Detection

Optimized Member Management¶

Read-only members no longer support viewing the member management page.
Supports filtering regular members and SSO members, SSO members only display members corresponding to the currently configured identity provider (IDP).

For more details, refer to Member Management.

Added DEMO Workspace¶

TrueWatch provides a workspace Demo to help you better use various functional modules of TrueWatch. You can click 'Workspace' in the upper left corner, then click 'Experience Demo Space' in the pop-up dialog to enter the Demo space for viewing.

Other Feature Optimizations¶

Chart metric aggregation function changed from default last to avg, log data aggregation function changed from default last to count
Optimized time series chart, pie chart legend copy experience
Optimized note edit mode interaction display
Snapshot supports saving current explorer display column information
Trace detail page formats time-related fields, converting timestamps to date format display
Deployment Plan management backend supports modifying workspace data retention policies

DataKit Updates¶

Breaking changes¶

Gitlab and Jenkins collectors adjusted CI/CD data-related time fields to unify frontend page data display effects

Collector Function Adjustments¶

Optimized IO module data processing, improved data throughput efficiency
Added disk caching function to various Trace data types
DataKit's own measurement adds metrics related to goroutine usage (datakit_goroutine)
MySQL collector adds mysql_dbm_activity measurement
Added netstat collector
TDengine adds log collection
Optimized disk collector fstype filtering, by default only collects common file systems
Log collector, for each log, adds field message_length representing current log length, facilitating log filtering by length
CRD supports locating Pod scope via DaemonSet
eBPF removes go-bindata dependency
Container collector by default opens k8s and container related metrics, which consumes additional time series to some extent

Bug Fixes¶

Fixed DataKit's own CPU usage rate calculation error
Fixed SkyWalking middleware identification issue
Fixed Oracle exit issue
Fixed Sink DataWay failure issue
Fixed HTTP /v1/write/:category interface JSON write issue

Documentation Adjustments¶

Almost every chapter added jump tags for permanent referencing by other documents
pythond documentation moved to Custom Development directory
Collector documentation migrated from original 'Integration' to 'DataKit' documentation library
DataKit documentation directory structure adjusted, reduced directory levels
Almost every collector added k8s configuration entry
Adjusted document header display, besides OS identifier, added election identifier for collectors supporting election

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Cloud Native
- Using CRD to Start Your Ingress Observability Journey
Monitoring
- Application Performance Monitoring (APM) - DDtrace Custom Instrumentation
- Application Performance Monitoring (APM) - DDtrace TrueWatch Secondary Development Practice

For more best practices updates, refer to Best Practices Version History.

August 18, 2022¶

TrueWatch Updates¶

TrueWatch Billing Update¶

TrueWatch billing items APM, RUM, Log added data retention policies and corresponding unit prices. Log data added 7-day data retention policy, APM and RUM added 3-day data retention policy. You can adjust corresponding data retention policies based on actual needs to save cost. For more details, refer to the document Billing Methods.

Optimized Explorer¶

Added Filter History¶

TrueWatch added explorer filter history function, supports current user saving explorer key:value search condition history in local browser. In the current workspace, you can directly use historical search conditions in different explorers, helping avoid repeated filter condition configuration.

Open filter history: Supports clicking the expand icon in the lower right corner of the explorer, or directly using shortcut keys (Mac OS: shift+cmd+k / Windows: shift+ctrl+k) to quickly open filter history.
Close filter history: Click the close button x or use the esc key to close filter history.

For more details, refer to the document Filter History Instructions.

Added Quick Filter Value Sorting¶

In explorer quick filters, added support for field values to be displayed sorted by percentage. Click the 'Settings' button in the upper right corner of the quick filter item, select 'Query Value TOP 5', to view the top five field attribute value statistics percentages for the current filter item. On the right side of the top list, supports clicking 'Positive Filter', 'Negative Filter' buttons to perform data filter queries in key:value form for the currently ranked field attribute values. For more explorer quick filter details, refer to the document Quick Filter Instructions.

In explorer time widget, added time widget input format hint page, optimized time widget text input box, supports real-time interaction with hint box, helping users quickly understand time widget supported input format types. For more explorer time widget details, refer to the document Time Widget Instructions.

Added Display Column Field Separator Line and Text Hint¶

When adding display columns in explorer, added field separator line and its addition text hint. When entering fields in display columns, preset fields are distinguished from existing fields by separator line, with hint text 'Create and Add', supports selecting and adding display columns via keyboard up/down arrow keys (↑ ↓). For more explorer display column details, refer to the document Display Column Instructions.

Added Scene Dashboard/Note/Explorer View Permissions.¶

When creating Dashboard, Note, Explorer in a scene, supports creator customizing view permissions to prevent creator's own dashboard from being accidentally modified, or prematurely publishing dashboards / notes / explorers that are not yet completed or not convenient to publish. Permission settings include 'Public' and 'Visible Only to Self'.

Public: Open to all members in the current workspace, other members' view and edit permissions are unaffected.
Visible Only to Self: Only visible to the creator, other members do not have view permission.

Note: When accessing non-public dashboards/notes/explorers via links, etc., non-creators cannot see them, prompting user has no permission to view.

Added Snapshot View Permissions¶

When saving snapshots in a scene or explorer, supports customizing view permissions. Permission settings include 'Public' and 'Visible Only to Self'.

Public: Means users in the current workspace can view the saved snapshot.
Visible Only to Self: Means except the current user, other users cannot view the saved snapshot.

After saving a snapshot in explorer, supports viewing historical snapshot time range, filter conditions. Time range is divided into 'Absolute Time', 'Relative Time', and 'Default' based on snapshot save selection. For more snapshot usage details, refer to the document Snapshot.

Optimized Monitors and Events¶

Added Monitor Test¶

Monitor added manual trigger monitor detection. If current detection rules trigger, you can view related details in the event explorer.

Optimized Intelligent Inspection Information Display¶

Intelligent inspection list added 'Data Scope', 'Detection Frequency', 'Grouping', 'Last Trigger Time', 'Trigger Status'. Operations added 'Export JSON Configuration', export file name format: Intelligent Inspection Name.json.

Intelligent inspection event detail page added 'Basic Attributes', 'History Records', 'Associated Events', and other common pages.

Optimized Event Detail Page¶

On event detail page, optimized 'Basic Attributes' page; 'Copy Current Event Content' adjusted to 'Export Event JSON' button, also added 'View Monitor Configuration' button on the left, supporting one-click jump to current event's monitor configuration page, convenient for real-time monitor rule adjustment.

If monitor configured alerts, then associated event detail page adds 'Alert Notification' page. If monitor set alert silence, during alert silence period, same alert events will not be sent to notification targets. On event detail page's 'Alert Notification' page, that alert notification is marked as 'Notification Not Sent'.

Other Feature Optimizations¶

TrueWatch added support for email verification method authentication
Explorer detail page associated network page optimized host, Pod, Deployment type display
Dashboard and built-in views, etc., places adding charts added switch icon, adjusted view variable edit button position
Time series chart legend value added sum summation statistics, also optimized legend display and interaction
Monitor, chart query log data filter conditions added wildcard and not wildcard

DataKit Updates¶

New Features¶

Pipeline added reftable function
DataKit 9529 HTTP supports binding to domain socket
- Corresponding eBPF collection and Oracle collection configuration methods also need corresponding changes.
RUM sourcemap added Android R8 support
CRD added log configuration support
- Complete example

Optimizations¶

Optimized container collector documentation
Added Common Tag documentation
Optimized election configuration and some related naming
Election collectors, when DataKit election is enabled, still support turning off election function on specific collectors
Supports specifying data type io block configuration
DDTrace collector sampling added meta information identification
DataKit's own measurement added 9529 HTTP request related metrics
Optimized Zipkin collection memory usage
DDTrace collector, after enabling disk caching, defaults to blocking IO feed
eBPF added process name (process_name) field
DCA new version released
Log HTTP data writes (logstreaming/Jaeger/OpenTelemetry/Zipkin) all added queue support
Log collection added automatic multi-line support

Bug Fixes¶

Fixed MySQL collector connection leak issue
Fixed Pipeline Json value extraction issue
Fixed macOS ulimit setting invalid issue
Fixed sinker-Dataway invalid in Kubernetes issue
Fixed HTTP data write interface data validation issue
Fixed eBPF collector struct offset calculation failure after kernel change issue
Fixed DDTrace close-resource issue

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Monitoring
- Using extract + TextMapAdapter to implement custom traceId
Insight
- Scene - Intranet Scenario Dubbo Microservice Integration with TrueWatch

For more best practices updates, refer to Best Practices Version History.

August 4, 2022¶

TrueWatch Updates¶

DCA Web Version Launched¶

TrueWatch added DataKit's web management tool DCA, designed to conveniently manage installed and configured collectors, supporting viewing collector operation status, collector configuration management, Pipeline management, blacklist management, and collector documentation help, etc. For more details, refer to the document DCA.

Explorer Search¶

In explorer search bar, added 'Exist', 'Not exist' filter logic, returning search results by judging whether filter fields exist.
In explorer search bar, added operation methods for numeric types, supporting operators including equal to, greater than or equal to, less than or equal to, greater than, less than, not equal to, etc.
Added field data type hints, matching mode description

For more explorer search details, refer to the document Search Instructions.

Quick Filters¶

In quick filter items, added label field search function. When quick filter items exceed 10 label fields, supports fuzzy search by 'Field Name' or 'Display Name'.
In quick filter items added 'Settings' button, supports adding / removing display columns.
In quick filter items added 'Settings' button, custom added personal-level filter item fields support editing display name and deleting fields.

For more explorer quick filter details, refer to the document Quick Filter Instructions.

Added interval display, supports clicking time widget display box to manually input time range, including 'Interval Range', 'Timestamp Range', and 'Standard Time Format'.
Dropdown list added custom options, custom range supports hour:minute:second format input.
Added recent 20 historical absolute time record viewing.

For more explorer time widget details, refer to the document Time Widget Instructions.

Display Columns¶

In explorer list, when mouse hovers over display column, added 'Settings' button, supports operations on display column like ascending, descending, move column left, move column right, add column left, add column right, replace column, add to quick filters, add to grouping, remove column, etc. For more explorer display column details, refer to the document Display Column Instructions.

Optimized Explorer Detail Page¶

Adjusted extended field quick filter 'Filter Field Value', 'Reverse Filter Field Value', 'Add to Display Column', and 'Copy' operation positions.
Supports switching between previous/next log details via keyboard up/down arrow keys (↑ ↓).
Supports directly closing detail page popup via keyboard esc button.
Added 'View Context' function: When viewing context on log detail page and opening a new log explorer, it will use 'host', 'source', 'service', 'filename' as filter conditions, with current log time as start time, current log time + 30 minutes as end time brought to time widget for context log filtering. For more details, refer to the document View Context.

Added Global Explorer Auto-Refresh Configuration¶

In TrueWatch workspace, click 'Account', can enable / disable 'Explorer Auto Refresh'.

Enable: Explorer data auto-refreshes according to time widget default data refresh time of 30 seconds, e.g., if selecting last 15 minutes, displays data for the last 15 minutes refreshed every 30 seconds.
Disable: Explorer time widget enters with 30-second auto-refresh turned off, e.g., if selecting last 15 minutes, displays content data for that 15-minute absolute time and no longer auto-refreshes, can click 'Play' button to refresh and view last 15 minutes data.

Note: Explorer auto-refresh only takes effect for local browser. For more explorer time widget details, refer to the document Time Widget Instructions.

Added Global Blacklist Function¶

TrueWatch supports filtering out different types of data meeting conditions by setting a blacklist, i.e., after configuring blacklist, data meeting conditions is no longer reported to the TrueWatch workspace, helping you save data storage costs. Global blacklist function currently supports filtering data types including Log, Basic Object, Resource Catalog, Network, APM, RUM, Security Check, Event, Metrics, Profile. For more details, refer to the document Blacklist.

TrueWatch supports manually setting function menus, including hide/show and adjusting menu order, helping you customize current workspace menu display. In TrueWatch workspace, click account in lower left corner, select 'Function Menu Management', then adjust menus. For more details, refer to the document Function Menu Management.

Added Chart Query Alias¶

In scene visualization charts, added chart query alias. After adding alias, legend name also changes, making it easier to intuitively distinguish related metrics. Currently supported charts include time series chart, pie chart, bar chart, scatter plot, bubble chart, funnel chart.

Added Time Series Chart, Pie Chart Legend Styles¶

In scene visualization charts, pie chart added support for legend styles including bottom, right, surround, hide. Time series chart supports legend styles including bottom, right, and hide, hidden by default.

Optimized Object Historical Data Retention Policy¶

TrueWatch stores 48-hour historical data for all objects (host, container, pod) reported to the workspace, convenient for users to review object data performance at a specific moment within the past two days. When viewing a log, trace, security inspection data, you can view the corresponding object situation at that moment in the associated host, container, Pod's 'Attribute View'.

Adjusted Save Snapshot Location¶

Removed 'Save Snapshot' button from explorer top navigation bar, uniformly migrated to 'History Snapshots'. Scene dashboards and notes can create snapshots via shortcut keys. For more details, refer to the document Snapshot.

Other Feature Optimizations¶

Time series chart time interval added down to millisecond level
Management backend added workspace-level index configuration adjustment entry
Log explorer distribution chart added support for custom selecting time interval
RUM explorer page added current data extended field page display

DataKit Updates¶

Some data types, after sending failure, support caching to disk, sending later
Supports sending data meeting conditions to different workspaces via different dataway addresses
Sourcemap added Android and iOS support
Container collector related updates:
- Fixed Kubernetes Node host OS information collection error
- Kubernetes Prom collection no longer automatically appends pod related information, avoiding time series explosion
- Pod object appended corresponding yaml information
Pipeline related updates:
- Optimized Pipeline execution steps
- grok() and json() functions by default perform trim-space operation
DDTrace related updates:
- Fixed potential goroutine leak issue
- Supports configuring disk cache to alleviate memory usage issues
Other Bug fixes:
- Optimized line protocol construction
- Log collection, removed periodic tail data cleanup function to alleviate possible log truncation issues

For more DataKit updates, refer to DataKit Version History.

July 21, 2022¶

TrueWatch Updates¶

Added Intelligent Inspection Function¶

Intelligent inspection is based on TrueWatch's intelligent detection algorithms, supports automatic detection of infrastructure and application problems, helping users discover issues occurring during IT system operation. Through root cause analysis, quickly locate abnormal problem causes. Through TrueWatch's intelligent prediction algorithms, helps users foresee potential infrastructure and application problems in advance, assess impact level of problems on system operation, better determine troubleshooting priority, reduce uncertainty in troubleshooting process.

Intelligent inspection currently supports three inspection templates:

Memory Leak: Detects whether hosts in current workspace have memory leak issues.
Disk Usage Rate: Detects whether hosts in current workspace have excessively high disk usage rate issues.
Application Performance Detection: Detects whether service QPS, average response time, P90 response time, and error rate in current workspace have fluctuation changes.

Optimized Explorer Search and Filter Functions¶

Explorer search added 'not wildcard reverse fuzzy match', i.e., not matching fuzzy query results.

Explorer quick filters added workspace-level and personal-level filtering methods
Workspace-level filter items: Configured by administrator/owner. Click 'Settings' button next to quick filters to configure workspace-level filter items.
Personal-level filter items: All members can configure quick filter items based on local browser. Click 'Edit' on the right of quick filters to configure personal-level filter items.

For more details, refer to the document Explorer Search and Filter.

Added Modifying Time Range in URL for Data Query¶

TrueWatch supports directly modifying the time parameter time range in the browser URL for current workspace explorer/dashboard data query, supports 4 units: second, minute, hour, day, e.g., time=30s, time=20m, time=6h, time=2d, etc. As shown below, modifying time=2h in browser, explorer displays last 2 hours data. For more details, refer to the document URL Time Range.

Added Dashboard View Variable Log, APM, RUM, Security Check Data Source Configuration¶

Dashboard view variables added 'Log', 'APM', 'RUM', 'Security Check' data source configuration, also optimized dashboard new and edit view variable interaction.

In dashboards without added view variables, top navigation bar adds 'Add View Variable' button, click to add view variable.
If dashboard already has view variables added, view variable names are directly displayed below top navigation bar. Click 'Edit' button on the right to enter view variable edit page.

For more details, refer to document View Variables.

Optimized Chart Query Interaction¶

Click the 'separator bar' between chart and query to drag up and down.
Click the 'drag' button on the left of query statement to adjust query order by dragging up and down.
Scroll bar only acts on query statement area, does not affect chart.

Added RUM Metric Detection Event Notification Template Variables¶

When configuring notification events in 'RUM Metric Detection' monitor, besides general template variables, additionally supports the following template variables. For more details, refer to document Event Title, Content Template.

Template Variable	Type	Description
`app_id`	String	Application ID
`app_name`	String	Application Name
`app_type`	String	Application Type

Optimized Event Content One-Click Open Link¶

When configuring monitor, add link in event content. After event triggers, on event detail page's 'Event Content', click link to open new page and jump.

DataKit Updates¶

prom collector built-in timeout duration is 3 seconds
Log related issue fixes:
- Added log collection log_read_offset field
- Fixed bug where log file after rotate not correctly readAll
Container collection related issue fixes:
- Fixed incompatibility issue with environment variable NODE_NAME
- k8s auto-discovered prom collector changed to serial, node distributed collection
- Added log source and multi-line mapping configuration
- Fixed bug where container log after replacing source still used previous multiline and pipeline
- Corrected container log, set file active duration to 12 hours
- Optimized docker container log image field
- Optimized k8s pod object host field
- Fixed issue where container metrics and object collection didn't add host tag
eBPF related:
- Fixed uprobe event name naming conflict issue
- Added more environment variable configurations for cloud k8s environment deployment
Optimized APM data receiving interface data processing, alleviating client deadlock and memory usage issues
SQLServer collector fixes:
- Restored TLS1.0 support
- Supports filtering collection via instance to reduce time series consumption
Pipeline function adjust_timezone() adjusted
IO module optimization, improved overall data processing capability, kept memory consumption relatively controllable
Monitor updates:
- Fixed possible long-term lag caused by Monitor during busy times
- Optimized Monitor display, added IO module information display for user IO module parameter adjustment
Fixed Redis crash issue
Removed some complex redundant logs
Fixed election collectors not appending host tag in non-election mode issue

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Cloud Platform Integration
- Rancher Deployment DataKit Best Practices
- Tencent Cloud Product Observability Best Practices (Function)

For more best practices updates, refer to Best Practices Version History.

July 7, 2022¶

TrueWatch Updates¶

Optimized Explorer Positive Selection, Negative Selection, Fuzzy Match Three Filter Modes¶

TrueWatch explorer supports exact filtering, fuzzy filtering (wildcard), and negative selection by 'field:value' method, supports editing and modifying filtered 'field:value'. For more details, refer to document Explorer Search and Filter.

Optimized Explorer Quick Filters¶

In TrueWatch explorer quick filters, select all by default, supports 'Positive Selection', 'Negative Selection', 'Reset', 'All', and 'Only' multiple filters. If search box includes both positive and negative selection, corresponding quick filter tags are grayed out and unselectable. For more details, refer to document Quick Filters.

Added Explorer Display Column Multiple Quick Operations¶

In TrueWatch explorer, via 'Display Columns', you can add, delete display columns to explorer list.

When mouse hovers over added display column, supports operations like ascending/descending order, move column left, move column right, remove column, etc., on added display column.
If display column content display incomplete, supports 'double-clicking separator line' with mouse on right side of display column to expand that column's content.

Optimized Explorer Detail Page Attribute/Field Quick Filter¶

On explorer detail page, when mouse clicks attribute field like 'Host', 'Source', supports 'Positive Filter', 'Negative Filter', 'Add to Display Column', and 'Copy' for quick filter viewing. In log explorer detail page, when mouse selects extended field, displays 'Copy', 'Positive Filter', 'Negative Filter', and 'Add to Display Column' icons for quick filter viewing.

'Copy', copies field to clipboard.
'Positive Filter', adds field to explorer, views all data related to that field.
'Negative Filter', adds field to explorer, views data other than that field.
'Add to Display Column', adds field to explorer list for viewing.

Optimized History Snapshot Function, Supports Three Time Save Policies¶

In Metrics, Log, Event, APM, RUM, Synthetic Testing, Security Check, CI Visualization, etc., explorers, you can click 'Save Snapshot' to save data content currently displayed by explorer, supports choosing to enable/disable 'Time Filter'.

Snapshot with time filter enabled, saves according to time selected on current page, shared link does not support switching time widget.
Snapshot with time filter enabled, and absolute time locked, converts current page selected time range to 'absolute time' for saving.
Snapshot with time filter disabled, follows system default time, shared snapshot can switch time widget.

After snapshot saved, can view in explorer history snapshot list. Select and open historical snapshot, click 'Return to Explorer' in upper right corner to return to default explorer.

Added Pipeline One-Click Get Sample Test Data¶

When creating or editing Pipeline, added 'One-Click Get' button on right of sample parsing test, convenient for you to get data for parsing debugging.

Log data: Returns message data.
Other data types: Returns line protocol format data.

For more details, refer to document Text Processing Pipeline.

Added Scene Custom Explorer Text Analysis Mode¶

In scene custom explorer, when editing explorer, in custom configuration display columns, supports selecting 'Approximate Text Analysis' explorer field, defaults to using message field corresponding content for approximate text analysis. For more details, refer to Scene Custom Explorer.

Added Log Explorer Detail Page Associated Network Pod and Deployment View¶

TrueWatch supports selecting to view host, Pod, and Deployment view data connection situations in log explorer detail page's 'Network'.

Note: Viewing related network in log detail requires matching corresponding fields.

Added Explorer Detail Page Associated Network 48-Hour Data Playback Function¶

In Log, Process, Trace, Host, Pod explorer detail page associated network, supports clicking time widget to select playback of 48-hour network data.

Time range: Default views data 30 minutes before and after; if currently occurring, defaults to viewing last 1 hour data.
Supports arbitrarily dragging time range to view corresponding network traffic.
After dragging, click 'Play' button or refresh page, returns to viewing 'Last 1 Hour' network data.

Adjusted Unrecovered Event Save Policy, Supports Manual Event Recovery¶

In unrecovered event list, mouse over event, can see 'Recovered' function on right of event. Click 'Recovered' to recover event, simultaneously generates a recovery event, corresponding operator can be viewed in that event.

Other Feature Optimizations¶

Chart locked time added [Latest 5 minutes] time range, time interval added [5s] [10s] [30s] three second-level time selections.
Scene explorer display columns, view variable basic object field attributes/tags support custom input.
Adjusted event detail associated dashboard position.
Added time series count statistics in metric management.
Optimized log detail page associated trace, displays flame graph and selects all data of corresponding span based on trace_id and span_id in log.
Optimized RUM service display and interaction.
RUM, Network, Synthetic Tests, CI explorer dropdown options adjusted to tiled display.
Monitor event notification content supports configuring template variable field mapping, supports configuring template variable corresponding values via DQL query statements.
Help center homepage added key function quick jump entry.

DataKit Updates¶

Adjusted global tag behavior to avoid tag splitting for election collectors.
SQLServer collector added election support.
Line protocol filter supports all data types.
9529 HTTP service added timeout mechanism.
MySQL
- dbm measurement name adjustment.
- service field conflict issue.
Container object added field container_runtime_name to distinguish different levels of container names.
Redis adjusted slowlog collection, changed its data to log storage.
Optimized TDEngine collection.
Improved Containerd log collection.
Pipeline added Profile data support.
Container/Pod log collection supports additionally appending tags on Label/Annotation.
Fixed Jenkins CI data collection time precision issue.
Fixed Tracing resource-type value inconsistency issue.
eBPF added HTTPS support.
Fixed possible log collector crash issue.
Fixed prom collector leak issue.
Supports configuring io disk cache via environment variables.
Added Kubernetes CRD support.

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Integration
- Skywalking Collection JVM Observability Best Practices
- Minio Observability Best Practices

For more best practices updates, refer to Best Practices Version History.

June 21, 2022¶

TrueWatch Updates¶

TrueWatch Help Documentation Fully Launched¶

To improve TrueWatch help documentation reading experience, TrueWatch help documentation has been migrated under TrueWatch domain. You can view TrueWatch help documentation more simply and quickly. New TrueWatch help documentation address: https://docs.truewatch.com.

Added Profile Observability¶

Profile supports collecting dynamic performance data during application runtime in different language environments like Java / Python, etc., helping users view CPU, memory, IO performance issues. Collecting profile data requires installing DataKit first and configuring Profile collector. After configuration, DataKit uploads collected profile data to your TrueWatch workspace. You can understand your program code performance via Profile real-time data explorer. For more details, refer to document Profile.

Pipeline Covers Full Data Text Analysis Processing¶

Text Processing (Pipeline) is used for data parsing, cutting various data types into structured data meeting our requirements by defining parsing rules. In TrueWatch workspace 'Management' - 'Text Processing (Pipeline)', click 'New Pipeline' to create a new pipeline file. For more details, refer to document Text Processing (Pipeline).

Added Deployment Network Details and Network Distribution¶

Deployment Network supports viewing network traffic between Deployments. Supports viewing network traffic and data connection situations from source IP to target IP based on IP/port, real-time display via visualization, helping enterprises understand business system network operational status in real time, quickly analyze, trace, and locate problem faults, prevent or avoid business problems caused by network performance degradation or interruption.

After Deployment network data collection succeeds, reported to TrueWatch console. In 'Infrastructure' - 'Containers' - 'Deployment' detail page's 'Network', you can view all Deployment network performance monitoring data information in workspace. For more details, refer to Deployment Network.

Optimized Event Detection Dimension Jump to Other Explorers¶

In unrecovered event explorer, supports clicking detection dimension to view related container, process, log, trace, RUM, synthetic testing, security check, CI, etc. If related explorer has no related data, corresponding jump link is grayed out and unclickable.

Added Log Explorer JSON Format Message Information Search¶

Log explorer added search for JSON format log content (message), search format: @key.key:value.

Note: JSON search only supports workspaces created after this feature launch.

Added RUM New Application Supports User Custom Input app_id Information¶

Added custom application ID function. Supports custom input app_id when using RUM new application function, generating a unique application ID identifier within current space, can be used to distinguish application types, data upload matching, etc.

Application Name (required): Used to identify current RUM implemented application name.
Application ID Identifier (optional): Unique application ID identifier within current space, supports customization, can be used for data upload matching. Application ID identifier up to 20 characters, only supports uppercase/lowercase letters.
Application ID: If application ID identifier filled when creating application, after clicking create application, filled application ID identifier information prepended to application ID, convenient for distinguishing applications for query and filtering.

Optimized Process Detection to Infrastructure Object Detection¶

Process detection optimized to Infrastructure Object Monitoring, added host, container, process, Pod, Deployment, Replicaset, Job, resource catalog, etc., basic object selection, used to monitor infrastructure object data within workspace. For more details, refer to document Infrastructure Object Detection.

Other Feature Optimizations¶

Infrastructure POD explorer heatmap mode added CPU usage rate, memory usage fill metrics.
Optimized log blacklist configuration. Supports manual input log source as log blacklist source.
Optimized APM service list data query time component, supports custom time range selection.
Optimized DataKit installation guide text on K8S, configuring DataWay data gateway address automatically adds current workspace token.
Optimized monitor configuration UI style.

DataKit Updates¶

2022/06/21¶

gitrepo supports passwordless mode.
prom collector
- Supports log mode collection.
- Supports configuring HTTP request headers.
Supports container log collection exceeding 16KB length.
Supports TDEngine collector.
Pipeline
- Supports XML parsing.
- Remote debugging supports multiple data types.
- Supports Pipeline calling external Pipeline scripts via use() function.
Added IP library (MaxMindIP) support.
Added DDTrace Profile integration.
Containerd log collection supports filtering rule configuration via image and K8s Annotation.
Documentation library overall switch.

2022/06/16¶

Log collection supports recording collection position, avoiding data omission due to DataKit restart, etc.
Adjusted Pipeline settings when processing different data types.
Supports receiving SkyWalking metric data.
Optimized log blacklist debugging function:
- Monitor displays filtered points count.
- Adds a .filter file under datakit/data directory to record fetched filters.
Monitor adds DataKit open file count display.
DataKit compiler upgraded to golang 1.18.3.

2022/06/07¶

Added TCP/UDP port detection collector.
DataKit and DataWay added DNS detection, supports DataWay DNS dynamic switching.
eBPF L4/L7 traffic data added k8s deployment name field.
Optimized OpenTelemetry metric data.
ElasticSearch added AWS OpenSearch support.
Line protocol limit, string length limit relaxed to 32MB.
prom collector added extra configuration, supports ignoring specified tag=value matches to reduce unnecessary time series.
Sink added Jaeger support.
Kubernetes related metric collection, all turned off by default to avoid time series explosion.
DataKit Monitor adds dynamically discovered (e.g., prom) collector list refresh.

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

APM
- GraalVM and Spring Native Project Achieving Trace Observability
Integration
- Host Observability Best Practices (Linux)

For more best practices updates, refer to Best Practices Version History.

Integration Template Updates¶

New Documents¶

Alibaba Cloud
- Alibaba Cloud NAT
- Alibaba Cloud CDN

New Views¶

Alibaba Cloud
- Alibaba Cloud NAT
- Alibaba Cloud CDN

June 6, 2022¶

TrueWatch Billing Update¶

TrueWatch billing optimized Time Series billing logic, and metric data Data Retention Policy. Original every 300 for 3 yuan lowered to every 1000 for 3 yuan. Also metric data added 3-day, 7-day, 14-day data retention policies, measurement supports Custom Data Retention Policy.

Time series counts all possible combinations based on tags that can be formed from reported metric data in current workspace. Data retention policy is data storage duration, how long data reported to current workspace is saved, data exceeding storage duration automatically deleted.

Time series is full count, i.e., daily generated time series accumulate sequentially during data retention policy period, longer data retention policy, higher time series cost.

This optimization by lowering time series cost and shortening metric data retention policy, you can more flexibly adjust metric data retention policy, helping you save cost.

TrueWatch Updates¶

Added Jenkins CI Observability¶

TrueWatch added Jenkins CI observability. You can directly view CI results in Jenkins via TrueWatch's CI visualization function. CI process is continuous integration. When developers push code, if encountering problems, can view all CI pipelines and their success rate, failure reasons, specific failure links in TrueWatch, helping provide code update guarantee. For more details, refer to CI Visualization.

Added Custom Explorer Chart Synchronized Search¶

Custom explorer added chart synchronized search switch, used to decide whether search conditions affect chart query, enabled by default. When search box has content, turn off switch, chart query returns to default state; turn on switch, chart query affected by filter content.

Added Network Topology and Service Topology Drill-down Analysis¶

In infrastructure network topology map, click host/Pod icon, click 'View Upstream/Downstream', can view current node's upstream/downstream node associations. In upstream/downstream nodes, click 'Return to Overview' in upper left returns to original network topology map, searching or filtering in search box filters associated upstream/downstream nodes, displays matching associated upstream/downstream nodes based on search or filter results. For more details, refer to document Network View Upstream/Downstream.

In service topology map, click service icon, click 'View Upstream/Downstream', can view current service's upstream/downstream service associations. In upstream/downstream services, click 'Return to Overview' in upper left returns to original service topology map, searching or filtering in search box filters associated upstream/downstream services, displays matching associated upstream/downstream services based on search or filter results. For more details, refer to document Service View Upstream/Downstream.

Added Delete Resource Catalog Data and Index¶

TrueWatch supports owners and administrators deleting specified resource catalog classifications and all resource catalogs. Enter 'Management' - 'Basic Settings', click 'Delete Resource Catalog', then select resource catalog deletion method, can delete corresponding object data.

Specified Resource Catalog Classification: Only deletes data under selected object classification, does not delete index.
All Resource Catalogs: Deletes all resource catalog data and index.

Note: Once all resource catalogs deleted, all data and index reported to infrastructure custom all deleted and unrecoverable, all set resource catalog classification data need re-reporting, limited to 5 delete all resource catalog operations per day.

For more resource catalog introduction, refer to document Custom.

Added Explorer Snapshot View Entry¶

In TrueWatch Metrics, Log, Event, APM, RUM, Synthetic Testing, Security Check, CI Visualization, etc., explorers, after saving snapshot, can directly click view snapshot icon in upper right to slide open and view saved snapshots.

Supports snapshot name keyword search, fuzzy matching related snapshot names via keyword.
First is default view, does not support share, copy link, delete functions.
Except first default view, other snapshots support share, copy link, delete functions, click 'Snapshot Name' opens corresponding data copy in current explorer.

Added Explorer Filter Condition Edit Function¶

In TrueWatch explorer search bar filtering via 'field:value' method, supports clicking 'field:value' to edit modify 'field:value', and filter with modified result.

Optimized RUM View Explorer Associated Trace to Fetch/XHR¶

In TrueWatch RUM View explorer detail page, switch to 'Fetch/XHR', supports viewing each network request sent to backend application during user access, including occurrence time, requested trace, and duration.

If network request has corresponding trace_id, prompt icon before request, click request, can jump to corresponding trace detail page.

Added Chart Data Loading High-Performance Mode¶

TrueWatch supports chart data loading high-performance mode, disabled by default, can enable by clicking account in lower left corner, select 'High-Performance Mode'. After high-performance mode enabled, all charts do not load dynamically, load simultaneously when entering page, i.e., charts already fully loaded when scrolling beyond current page view, can directly view display results.

Note: High-performance mode only affects current user viewing charts.

Added Alert Configuration Event Notification Level¶

Alert configuration supports custom selecting event notification levels, including Emergency, Important, Warning, Recovery, Data Gap, Data Gap Recovery, Data Gap Considered Recovery 7 choices, supports multi-select, supports one-click clear options, after clear need manually select corresponding values. For more alert configuration, refer to document Alert Settings.

Other Feature Optimizations¶

Scene dashboard combination chart supports hide/show main title.
Optimized event detail page event type text display.
Infrastructure list view added sort by field function.
Log explorer added hide distribution chart button.
Explorer supports searching display columns via keyword, supports custom display columns as preset fields, later after cutting fields via Pipeline and reporting data, directly display reported data.
In built-in template library and built-in views added one-click view corresponding integration documentation, helping you quickly configure corresponding collector.
Built-in views besides supporting binding trace service, application, log source, project, tag, etc., related views in explorer, added support for custom key and value binding related views, also supports service slide detail page binding built-in views.
Optimized notification target Lark robot, supports custom whether need secret key security verification.
When configuring monitor, if configured data scope less than detection frequency, triggers prompt configuration may have data gap issue.

DataKit Updates (2022/05/26)¶

Pipeline adjusted, all data types can be additionally processed via configuring Pipeline.
grok() supports directly extracting field as specified type, no longer need additional type conversion via cast() function.
Pipeline added multi-line string support, for very long strings (e.g., regex cutting in grok), can write them as multiple lines, improving readability.
Each Pipeline operation status can be directly viewed via datakit monitor -V.
Added Kubernetes Pod object CPU/memory metrics.
Helm added more Kubernetes version installation adaptation.
Optimized OpenTelemetry, HTTP protocol added JSON support.
DataKit when auto-correcting line protocol, added log records for correction behavior, convenient for debugging data issues.
Removed all string metrics from time series data.
In DaemonSet installation, if election namespace configured, for participating election collectors, their data all add specific tag (election_namespace).
CI observability, added Jenkins support.

Breaking changes¶

For Docker container log collection, need mount host (Node) /varl/lib path into DataKit (because Docker logs default under host /var/lib/), in datakit.yaml, add following configuration in volumeMounts and volumes:

volumeMounts:
- mountPath: /var/lib
  name: lib

# Omit other parts...

volumes:
- hostPath:
    path: /var/lib
  name: lib

Best Practices Updates¶

APM
- Based on TrueWatch, using SkyWalking to achieve RUM, APM, and log correlation analysis.
Monitoring Best Practices
- OpenTelemetry Observability Construction
- OpenTelemetry to Jaeger, Grafana, ELK
- OpenTelemetry to Grafana
- OpenTelemetry to TrueWatch
TrueWatch Tips
- OpenTelemetry Sampling Best Practices

Integration Template Updates¶

New Documents and Views¶

Data Collection
- Opentelemetry Collector
Container Orchestration
- Kubernetes Scheduler
- Kubernetes Controller Manager
- Kubernetes API Server
- Kubernetes Kubelet

New Views¶

Container Orchestration
- Kubernetes Nodes Overview
Middleware
- JVM Kubernetes

May 19, 2022¶

TrueWatch Updates¶

Optimized TrueWatch Commercial Plan Registration Process¶

TrueWatch registration provides Free Plan and Commercial Plan registration options. You can register TrueWatch account according to actual needs. TrueWatch supports pay-as-you-go, usage-based billing, providing you with out-of-the-box, comprehensive observability cloud platform.

Added Scene Dashboard User View Template Library¶

TrueWatch built-in over 60 system view templates, no configuration needed, select and use, meeting various monitoring scenario needs. You can customize views as user view templates to create dashboards with one click.

Added Scene Custom Explorer Log Source and Filter Linkage¶

Scene custom explorer data scope added filter function, based on log source, filter field data under that log source, supports matching multiple field data, combined with search and grouping can further filter log data.

Added Event Detail Page Content Copy as Json Format¶

On event detail page, supports clicking 'Copy Complete Event Content' button to obtain all key data corresponding to current event. If dashboard associated when configuring monitor, can click 'Associated Dashboard' button to jump to corresponding dashboard.

On event detail page's Associated Information, if 'Log Detection' configured multiple expression queries (same object type), associated information supports multiple expression query tab switching. If two expression queries A and B, then relationship information includes A and B two tabs for switching view.

Added Log Data Obfuscation Processing¶

TrueWatch added log data obfuscation processing. After data collection reported to TrueWatch workspace, some data may contain sensitive information, e.g., IP addresses, user information, etc. For this information can do obfuscation processing by configuring sensitive fields.
Note:

Obfuscated data only supports workspace administrator and above members to view, standard and read-only members cannot view obfuscated information.
Configuring sensitive fields only supports workspace administrator and above members to operate, standard and read-only members only support viewing configured sensitive fields.

Optimized Log Explorer and Detail Page¶

TrueWatch Log Explorer by default displays 'time' and 'message' fields. This optimization supports hiding 'message' field display.

On log detail page, log content automatically displays Json and text two view modes based on message type. If log has no message field, does not display log content part. Log content supports expand/collapse, expanded by default, after collapse only displays 1 line height.

Extended fields display all related fields of log, supports 'Copy' and 'Add to Filter' for quick filter viewing.

Added Network Data Detection Monitor¶

Network Data Detection used to monitor network performance metric data within workspace. By setting threshold range, when metric reaches threshold triggers alert. 'TrueWatch' supports setting alerts for single metric and custom alert levels. In Monitor, click '+ New Monitor', select 'Network Data Detection', enter detection rule configuration page.

Optimized Built-in View Binding Function¶

TrueWatch built-in views include system views and user views. This optimization cancels custom binding system views as explorer views, only supports binding user views as explorer views. If need bind system views, can first clone system view as user view. If system view and user view same name, explorer prioritizes displaying user view. For how to bind user views as explorer views, refer to document Bind Built-in Views.

Other Feature Optimizations¶

Billing added prepaid card balance.
Infrastructure detail style optimization.
Trace detail page attribute line break display optimization.
Monitor configuration template variable display optimization.
Added shortcut, DQL query and snapshot menu moved under shortcut.
TrueWatch management backend supplemented template management classification information.

DataKit Updates (2022/5/12)¶

eBPF added arm64 support.
Line protocol construction supports auto-correction.
DataKit main configuration added example configuration.
Prometheus Remote Write supports tag rename.
Merged community edition DataKit existing functions, mainly includes Sinker function and filebeat collector.
Adjusted container log collection, DataKit directly supports containerd container stdout/stderr log collection.
Adjusted DaemonSet mode hostname acquisition strategy.
Trace collector supports filtering resource (resource) via service name (service) wildcard.

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Cloud Native
- Using TrueWatch to Start Rancher Observability Journey
Microservice Observability Best Practices
- Kubernetes Cluster Application Using SkyWalking Collection Trace Data
- Kubernetes Cluster Log Reporting to Same Node DataKit Best Practices
Gitlab-CI Observability Best Practices
- Gitlab-CI Observability Best Practices

For more best practices updates, refer to Best Practices Version History.

Integration Template Updates¶

New Documents and Views¶

Middleware
- Resin
- Beats
Host System
- Procstat

New Views¶

Container Orchestration
- Istio Service
Alibaba Cloud
- ASM Service

May 6, 2022¶

TrueWatch Updates¶

Optimized TrueWatch Commercial Plan Upgrade Process¶

TrueWatch upgrade to Commercial Plan by default opens TrueWatch Billing Center Account Settlement, supports changing settlement method to cloud account settlement, including Alibaba Cloud Account and AWS Cloud Account settlement methods.

Added Process, Log, Trace Detail Page Associated Network¶

TrueWatch Process, Log, Trace detail pages added associated network data analysis, supports viewing network traffic and data connection situations from source host/source process service to target based on IP/port, real-time display via visualization, helping enterprises understand business system network operational status in real time, quickly analyze, trace, and locate problem faults, prevent or avoid business problems caused by network performance degradation or interruption.

Scene Module Optimization¶

Optimized Dashboard, Removed Edit Mode¶

In scene Dashboard top navigation bar, removed 'Edit' button, added 'Add Chart' to add new charts to dashboard. After chart added, click 'Complete Adding' in upper right corner.

In Chart, click 'Settings' button, select 'Modify', can edit chart.

Added Chart Link Display Switch¶

TrueWatch supports chart built-in links and custom links, can help you jump from current chart to target page. Built-in links are association links TrueWatch provides for charts by default, mainly based on current query time range and grouping tags, helping you view corresponding logs, processes, containers, traces. Built-in link display switch disabled by default, can enable when editing chart. Custom links after creation, display switch enabled by default. For more details, refer to document Chart Links.

Optimized DQL Query and Simple Query Conversion¶

Click switch button on right of DQL Query, can switch DQL query to simple query.
Note: When DQL Query switches to Simple Query, if cannot parse or parse incomplete:

Under Simple Query no operation, directly switch back to DQL Query displays previous DQL query statement.
Under Simple Query adjusted query statement, switch back to DQL Query parses according to latest Simple Query.

For more DQL query and simple query application, refer to document Chart Query.

Monitor and Event Module Optimization¶

Added Event Associated Information¶

TrueWatch supports viewing information related to triggering current event, e.g., viewing logs related to triggering event. This Associated Information only supports events generated by 4 monitors: Log Detection, Security Check Anomaly Detection, Process Anomaly Detection, and Synthetic Testing Anomaly Detection.

Added Data Gap Event Name and Content Configuration¶

TrueWatch monitor 'Threshold Detection', 'Level Detection', 'Range Detection', 'Mutation Detection', 'Process Anomaly Detection', 'APM Metric Detection', 'RUM Metric Detection' added data gap event title and content configuration, not fillable by default. When select trigger data gap event becomes fillable data gap event name, supports using preset template variables, details refer to Event Name/Content Template.

Optimized Synthetic Testing Anomaly Detection¶

TrueWatch monitor Synthetic Testing Anomaly Detection, optimized supports selecting HTTP, TCP, ICMP, WEBSOCKET testing types.

Optimized Alert Notification Template, Added Association Jump Link¶

Email, DingTalk, WeChat, Lark received alert notifications include 'TrueWatch Jump Link', click can directly jump to corresponding TrueWatch event detail, time range is 15 minutes before current time, i.e., event at 18:45:00, after clicking link jumps to event detail page, time range fixed to 4.20 18:30:00 ~ 4.20 18:45:00. For more alert notification, refer to document Alert Settings.

Other Feature Optimizations¶

Optimized service servicemap metric query performance.
Added explorer numeric field supports > | >= | < | <= | [] 5 writing methods.
Added metric explorer tag supports cascade filtering.
Optimized DQL query return error prompt.

DataKit Updates¶

Process collector filtering only affects metric collection, object collection unaffected.
Optimized DataKit sending DataWay timeout issue.
Optimized Gitlab collector.
Fixed log collection truncation issue.
Fixed various trace collectors reload partial configuration not taking effect issue.

For more DataKit updates, refer to DataKit Version History.

Integration Template Updates¶

New Data Storage Redis Sentinel Integration Document and View¶

Redis-sentinel observability scenario mainly displays Redis cluster, slaves, node distribution information, etc.

April 26, 2022¶

TrueWatch Updates¶

Optimized SSO Single Sign-On¶

TrueWatch supports users enabling SSO single sign-on based on workspace. Users during login input company email, obtain corresponding SSO login, achieve corresponding verification login. In TrueWatch workspace 'Management' - 'SSO Management' - 'Enable', can set SSO single sign-on for employees. This optimization mainly includes following points:

One workspace from supporting creating multiple identity providers updated to only support configuring one SSO single sign-on, by default your last updated SAML2.0 configuration considered final single sign-on verification entry.
If multiple workspaces configure same identity provider (IdP) data, after SSO single sign-on can switch view corresponding workspace data.
When configuring SSO single sign-on, 'User Whitelist' configuration replaced with 'Email Domain', only need configure email suffix domain to match user email suffix configured by identity provider (IdP) for single sign-on.
SSO single sign-on enable, configuration update, delete supports email notification and generates audit events.
SSO single sign-on users support delete and edit, edit can upgrade permission to 'Administrator'.

For more SSO single sign-on details, refer to document SSO Management.

April 21, 2022¶

TrueWatch Community Edition Launched¶

TrueWatch Community Edition provides teachers, students, cloud computing enthusiasts, etc., community users a simple, accessible, yet fully functional productized local deployment platform. Welcome to apply for free and download trial, build your own TrueWatch platform, experience complete product functionality.

TrueWatch Updates¶

Added Gitlab CI Observability¶

TrueWatch supports visualizing CI process and results built into Gitlab. You can directly view CI results in Gitlab via TrueWatch's CI visualization function. CI process is continuous integration. When developers push code, if encountering problems, can view all CI pipelines and their success rate, failure reasons, specific failure links in TrueWatch, helping provide code update guarantee. For more details, refer to CI Explorer.

Added Online Help Obus Assistant¶

TrueWatch Obus Assistant supports you quickly viewing basic introduction, advanced guide, best practices, DataKit, Func, etc., documents in workspace, by clicking provided keywords or directly inputting keywords in search bar for search, helping you quickly obtain related document instructions. For more details, refer to document Help.

Added Dashboard Set Refresh Frequency¶

TrueWatch supports setting refresh frequency in scene dashboard. Initial refresh frequency default 30 seconds, supports 10 seconds, 30 seconds, 60 seconds three options. If time widget 'Paused', then no longer refreshes.

Added Process 48-Hour Playback¶

TrueWatch infrastructure process supports viewing process data collected within last ten minutes. Click time range can view process 48-hour playback. After dragging, refresh pauses, time displays as: [ start time - end time ], query time range 5 minutes. Click 'Play' button or refresh page, returns to viewing 'Last 10 minutes' process.

Added Integration DataKit Kubernetes(Helm) Installation Guide Page¶

In TrueWatch integration DataKit installation guide page, added Kubernetes(Helm) installation guide, introduces how to install DataKit using Helm in K8S.

APM added global performance overview view. You can view online service count, P90 service response time, service maximum impact time, service error count, service error rate statistics on overview page. Also can view P90 service, resource, operation response time Top10 ranking, and service error rate, resource 5xx error rate, resource 4xx error rate Top10 ranking.

In APM service list, supports you clicking service type icon for category filtering, click again restores all view.

In service list, switch to 'Topology Map' mode can view call relationships between services. Supports drawing trace topology map based on service (service) and service environment version (service+env+version) two dimensions. After enabling 'Distinguish Environment and Version', draws service topology map according to different environment versions. For example, canary release, by enabling environment and version, can view service call situations under different environment versions.

Optimized SSO Single Sign-On Configuration¶

SSO single sign-on configuration user whitelist adjusted to email domain, used to verify if email suffix input at single sign-on matches, matching email can obtain SSO login link online. For more SSO configuration details, refer to document SSO Management.

Other Feature Optimizations¶

Added trace detail page associated log 'All Sources' option.
Added metric filter supports negative selection, aggregation function position adjustment.
Optimized log, APM, RUM, security check generate metric, 'Frequency' selected time also as aggregation period.
Optimized TrueWatch Deployment Plan workspace owner transfer permission function canceled, supports management backend setting.
Optimized alert notification SMS template.
Optimized synthetic tests new test list, supports directly selecting HTTP, TCP, ICMP, WEBSOCKET test.

DataKit Updates¶

Pipeline module fixed Grok dynamic multi-line pattern issue.
DaemonSet optimized Helm installation, added enable pprof environment variable configuration. DaemonSet all default enabled collectors each configuration supports configuration via environment variables.
Tracing collector preliminary supports Pipeline data processing.
Synthetic tests collector added failed task exit mechanism.
Log added unknown level (status), for logs not specifying level are unknown.
Container collector fixes:
- Fixed cluster field naming issue.
- Fixed namespace field naming issue.
- Container log collection, if Pod Annotation not specifying log source, then DataKit derives log source according to this priority.
- Object reporting no longer limited by 32KB length limit (because Annotation content exceeds 32KB), all Kubernetes objects remove annotation.

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Microservice Observability Best Practices
- service mesh microservice architecture from development to canary release full process best practices (part 1)
- service mesh microservice architecture from development to canary release full process best practices (part 2)
- service mesh microservice architecture from development to canary release full process best practices (part 3)
Monitoring Best Practices
- JAVA OOM Exception Observability Best Practices

For more best practices updates, refer to Best Practices Version History.

Integration Template Updates¶

New Documents¶

Application Performance Monitoring (APM)
- Node.JS
- Ruby
Middleware
- RocketMQ
Container Orchestration
- Istio
- Kube State Metrics
Data Storage
- Aerospike

New Views¶

Container Orchestration
- Kubernetes Overview by Pods
- Istio Mesh
- Istio Control Plane
Alibaba Cloud
- Alibaba Cloud ASM Mesh
- Alibaba Cloud ASM Control Plane
- Alibaba Cloud ASM Workload
Middleware
- RocketMQ

April 8, 2022¶

TrueWatch Billing Update¶

Added Alibaba Cloud Account Settlement Method¶

In TrueWatch Billing Center 'Manage Workspaces', supports viewing all workspaces bound under account, supports modifying settlement method of bound workspaces. Via 'Change Settlement Method' can arbitrarily switch between TrueWatch Billing Center account, Amazon Cloud account, and Alibaba Cloud account settlement.

TrueWatch Updates¶

Added DQL Query Explorer¶

DQL is language specifically developed for TrueWatch, simple syntax, convenient use, can perform data query via DQL language in TrueWatch workspace or terminal devices.

In TrueWatch workspace, click menu bar 'DQL Query' to open DQL query explorer, or you can directly open DQL query via shortcut Alt+Q. DQL query explorer supports table and JSON two return results, supports saving 7 days historical query records.

Synthetic Tests Added TCP/ICMP/Websocket Testing Protocols¶

TrueWatch supports custom testing tasks. By creating testing tasks based on HTTP, TCP, ICMP, WEBSOCKET, etc., different protocols, comprehensively monitors network performance, network quality, network data transmission stability, etc., from different regions, different operators to various services.

Added Infrastructure Network Module¶

In Infrastructure, original host network Map and Pod network Map migrated from host and container modules to new module 'Network', supports viewing host and Pod network Maps. Pod network Map fill metrics added layer 7 network metrics: requests per second, error rate, and average response time. For more details, refer to document Network.

Infrastructure Container Pod Added HTTP Layer 7 Network Data Display¶

Infrastructure container Pod added HTTP layer 7 network data collection and display. After Pod network data collection succeeds, reported to TrueWatch workspace. In 'Infrastructure' - 'Containers' - 'Pod' detail page's 'Network', you can view all Pod network performance monitoring data information within workspace. For more details, refer to document Pod Network.

Added Explorer Quick Filter 'Negative Selection' and 'Reset' Functions¶

In any explorer's 'Quick Filters', supports performing 'Negative Selection' or 'Reset' when selecting field filter content. 'Negative Selection' means selected field filter content not displayed (click 'Negative Selection' again returns field selected state). 'Reset' clears filter conditions. For more details, refer to Log Explorer Quick Filter.

Optimized Log Blacklist¶

TrueWatch supports filtering out logs meeting conditions by setting log blacklist, i.e., after configuring log blacklist, log data meeting conditions no longer reported to TrueWatch workspace, helping users save log data storage costs.

Log blacklist optimization supports matching all log sources, supports two blacklist configuration methods:

Meet any one filter condition, triggers blacklist filtering.
Meet all filter conditions, triggers blacklist filtering.

For more blacklist configuration, refer to document Log Blacklist.

Other Feature Optimizations¶

Added trace detail page span count statistics.
Optimized trace associated host time series drawing method.
Optimized overview chart time slicing, removed option. If previous overview chart enabled time slicing, after optimization default changed to not enable time slicing.
Optimized combination chart when browser zoomed, after entering edit cannot achieve combination chart switching edit different chart queries.
Optimized log explorer manual pause page refresh, scroll bar sliding to top not trigger auto-refresh.

DataKit Updates¶

Added host runtime memory limit, installation stage supports memory limit configuration.
CPU collector added load5s metric.
Supports TrueWatch optimized log blacklist function, adjusted monitor layout, added blacklist filtering situation display.
DaemonSet installation added Helm support, added DaemonSet installation best practices.
eBPF added HTTP protocol collection. For host installation, eBPF collector by default no longer installed, if need install requires specific installation command, DaemonSet installation unaffected.

For more DataKit updates, refer to DataKit Version History.

TrueWatch Mobile App Updates¶

Added site login capability, optimized scene, event explorer, maintained same access experience as web explorer.

Supports users selecting account corresponding site, login via account password or verification code.
Supports users viewing all sources or any source log data.
Supports users viewing all dashboards under current space, and via dropdown menu switch 'All Dashboards', 'My Favorites', 'Import Project', 'My Creation', and 'Frequently Viewed' to quickly filter find corresponding dashboard.
Supports users in Event explorer, via 'All' view, search, and filter all unrecovered event content triggered by anomaly detection library. Via 'My' events, view events currently still unrecovered notified to user via email, DingTalk robot, WeCom robot, Webhook, etc.

Best Practices Updates¶

TrueWatch Tips
- Multi-Microservice Project Performance Observability Practice
- ddtrace advanced usage
- Kubernetes Cluster Using ExternalName Mapping DataKit Service
Integration Best Practices
- OpenTelemetry Trace Data Integration Best Practices
Microservice Observability Best Practices
- Based on Alibaba Cloud ASM Achieving Microservice Observability Best Practices

For more best practices updates, refer to Best Practices Version History.

Integration Template Updates¶

New Alibaba Cloud PolarDB Oracle Integration Document, View, and Monitor¶

Alibaba Cloud PolarDB Oracle metric display, including CPU usage rate, memory usage rate, network traffic, connections, IOPS, TPS, data disk size, etc.

New Alibaba Cloud PolarDB PostgreSQL Integration Document, View, and Monitor¶

Alibaba Cloud PolarDB PostgreSQL metric display, including CPU usage rate, memory usage rate, network traffic, connections, IOPS, TPS, data disk size, etc.

New Alibaba Cloud RDS SQLServer Integration Document, View, and Detection Library¶

Alibaba Cloud RDS SQLServer metric display, including CPU usage rate, disk usage rate, IOPS, network bandwidth, TPS, QPS, etc.

New DataKit Integration Document, View, and Monitor¶

DataKit performance metric display, including CPU usage rate, memory information, uptime, log records, etc.

New Nacos Integration Document, View¶

Nacos performance metric display: Nacos online duration, Nacos config long connections count, Nacos config configuration count, Service Count, http request count, etc.

March 24, 2022¶

TrueWatch Site Update¶

TrueWatch supports multi-site login and registration, added 'International Region 1 (Oregon)' site. Original 'China Region 1 (Alibaba Cloud)' changed to 'China Region 1 (Hangzhou)'. Original 'China Region 2 (AWS)' changed to 'China Region 2 (Ningxia)'.

Accounts and data for different sites are independent, cannot share or migrate data between sites. You can choose suitable site for registration and login based on resource usage. Currently TrueWatch supports following three sites. For how to choose site, refer to document TrueWatch Site Instructions.

Site	Login URL	Operator
China Region 1 (Hangzhou)	https://auth.truewatch.com/	Alibaba Cloud (China Hangzhou)
China Region 2 (Ningxia)	https://aws-auth.truewatch.com/	AWS (China Ningxia)
International Region 1 (Oregon)	https://us1-auth.truewatch.com/	AWS (US Oregon)

TrueWatch Updates¶

Added Workspace Data Authorization¶

TrueWatch supports authorizing data from multiple workspaces to current workspace via data authorization method, querying and displaying via scene dashboard and note chart components. If multiple workspaces, after configuring data authorization, can view all workspace data in one workspace. For more configuration details, refer to document Data Authorization.

Configure workspaces needing authorized data view in 'Management' - 'Data Authorization'.
After workspace obtains data authorization, open 'Scene' - 'Dashboard or Note', select chart component, choose authorized workspace in 'Settings' 'Workspace', then can view and analyze authorized workspace data via Chart Query.

Added Save Online Pipeline Sample Test Data¶

TrueWatch Pipeline supports custom and official library two types:

After custom Pipeline script rule writing completed, can input log sample data for testing to verify if configured parsing rule correct. After custom Pipeline saved, log sample test data synchronously saved.
Pipeline official library comes with multiple log sample test data. Can choose log sample test data meeting own needs before 'Clone'. After cloned Pipeline modified saved, log sample test data synchronously saved.

For more online Pipeline function details, refer to document Pipelines.

Optimized Resource Catalog Explorer¶

In TrueWatch workspace, via 'Infrastructure' - 'Custom' - 'Add Object Classification', you can create new object classification, and resource catalog classification name and object fields.

After adding resource catalog classification, can perform custom data reporting via Func Function Processing Platform. For how to report data to TrueWatch workspace via Func, refer to document Resource Catalog Data Reporting.

Snapshot share supports setting validity time, supports choosing '48 hours' or 'Permanent'. In snapshot list, click share button, can perform advanced setting 'Hide Top Bar' in popup dialog. For more snapshot share details, refer to document Snapshot.

Note: Permanent share easily has data security risks, please use cautiously.

Optimized Chart Time Interval¶

In scene dashboard chart settings time interval select 'Auto Align', when previewing chart, time interval option appears in upper right corner of chart. You can choose time interval to view your data according to actual situation.

Optimized Process, APM, RUM Detection Data Gap Trigger Strategy¶

In TrueWatch monitoring function module, when configuring Process Anomaly Detection, APM Metric Detection, RUM Metric Detection monitors, data gap status supports 'Trigger Data Gap Event', 'Trigger Recovery Event', 'Do Not Trigger Event' three configurations, need manually configure data gap handling strategy.

Other Feature Optimizations¶

Optimized integration DataKit, Func installation guide pages.
Optimized log explorer single log complete display.
Added explorer associated search NOT combination.
Optimized edit member permission display.

DataKit Updates¶

Added DataKit command line completion function, helping you command prompt and parameter completion when operating in terminal.
Allows DataKit upgrade to non-stable version, experience latest experimental features. If you are production environment, please upgrade cautiously.
Preliminary supports Kubernetes/Containerd architecture data collection.
Network testing added TCP/UDP/ICMP/Websocket several protocol support.
Adjusted Remote Pipeline storage locally in DataKit, avoiding filename case issues caused by different file system differences.
Pipeline added decode() function, can avoid configuring encoding in log collector, achieve encoding conversion in Pipeline. add_pattern() added scope management.

For more DataKit updates, refer to DataKit Version History.

Best Practices Updates¶

Scene Best Practices
- RUM Data Reporting DataKit Cluster Best Practices
Log Best Practices
- Pod Log Collection Best Practices

For more best practices updates, refer to Best Practices Version History.

Integration Template Updates¶

New Alibaba Cloud PolarDB Mysql Integration Document, View, and Detection Library¶

Alibaba Cloud PolarDB Mysql metric display, including CPU usage rate, memory hit rate, network traffic, connections, QPS, TPS, read-only node delay, etc.

March 10, 2022¶

TrueWatch Billing Update¶

Added TrueWatch Billing Prepaid Card¶

TrueWatch prepaid card supports purchase via account cash balance, applicable to all TrueWatch consumption modes, including pay-as-you-go and annual subscription. Login to TrueWatch Billing Center, click 'Manage Prepaid Cards', enter prepaid card management page to purchase. After prepaid card purchased and payment made, invoice issued for equal amount based on actual payment. For more details, refer to Prepaid Card Management.

TrueWatch Updates¶

Added RUM resource (resource), action (operation), long_task (long task), error (error) Explorers¶

RUM explorer helps you view and analyze detailed information of user accessing application. Open 'RUM' in TrueWatch workspace, click any application then can understand each user session, page performance, resource, long task, errors in dynamic components, impact of latency on users via 'Explorer', help you comprehensively understand and improve application operational status and usage via search, filter, and correlation analysis, improve user experience.

TrueWatch RUM explorers include session (session), view (page), resource (resource), action (operation), long_task (long task), error (error).

Explorer Type	Overview
session (session)	View series details of user access, including user access time, access page path, access operation count, access path, and error information appearing, etc.
view (page)	View user access environment, trace user operation path, decompose user operation response time, understand performance metrics of series backend application calls caused by user operation
resource (resource)	View various resource information loaded on webpage, including status code, request method, resource address, load time, etc.
action (operation)	View user operation interactions during application usage, including operation type, page operation details, operation time, etc.
long_task (long task)	View long tasks blocking main thread exceeding 50ms during user application usage, including page address, task time, etc.
error (error)	View frontend errors browser issued during user application usage, including error type, error content, etc.

Added Pod Network Details and Network Distribution¶

Pod Network supports viewing network traffic between Pods. Supports viewing network traffic and data connection situations from source IP to target IP based on IP/port, real-time display via visualization, helping enterprises understand business system network operational status in real time, quickly analyze, trace, and locate problem faults, prevent or avoid business problems caused by network performance degradation or interruption.

After Pod network data collection succeeds, reported to TrueWatch console. In 'Infrastructure' - 'Containers' - 'Pod' detail page's 'Network', you can view all Pod network performance monitoring data information within workspace. For more details, refer to Pod Network.

In 'Infrastructure' - 'Containers' - 'Pod', click network distribution map icon in upper left corner, can switch to view Pod network distribution situation. In 'Network Distribution Map', you can visually query network traffic between Pods within current workspace, quickly analyze TCP delay, TCP fluctuation, TCP retransmission count, TCP connection count, and TCP close count between different Pods. For more details, refer to Pod Network Distribution Map.

DataKit Updates¶

DataKit collector added supports SkyWalking, Jaeger, Zipkin data configuration sampling strategy.
DataKit collector added supports OpenTelemetry data integration.
DataKit documentation library added document DataKit overall log collection introduction, including four ways: obtaining logs from disk files, obtaining logs via calling environment API, remotely pushing logs to DataKit, Sidecar form log collection.

Breaking Changes¶

2022/03/22

This time made major adjustments to Tracing data collection, involving several incompatibilities:
- DDtrace original conf configured ignore_resources field need change to close_resource, and field type changed from original array ([...]) form to dictionary array (map[string][...]) form.
- DDTrace original data collected tag **type** field changed to **source_type**

2022/03/04

Old version DataKit if enabled RUM function, after upgrade, need reinstall IP library, old version IP library cannot be used.

2021/12/30

Old version DataKit via datakit --version already cannot push new upgrade command, directly use following command:
Linux/Mac:

DK_UPGRADE=1 bash -c "$(curl -L https://static.truewatch.com/datakit/install.sh)"

Windows

$env:DK_UPGRADE="1"; Set-ExecutionPolicy Bypass -scope Process -Force; Import-Module bitstransfer; start-bitstransfer -source https://static.truewatch.com/datakit/install.ps1 -destination .install.ps1; powershell .install.ps1;

For more DataKit updates, refer to DataKit Version History.

SDK Updates¶

RUM compatible with Opentracing protocol trace tools. Web, mini program, Android, iOS SDK supports OTEL, SkyWalking, Jaeger, etc., trace tool data linkage.

Best Practices Updates¶

Custom Integration Best Practices
- Quick Start pythond Collector Best Practices
- Alibaba Cloud 'Cloud Monitor Data' Integration Best Practices
Log Best Practices
- logback socket log collection best practices

For more best practices updates, refer to Best Practices Version History.

Scene Template Updates¶

New Scene Custom Explorer MySQL Database Explorer Template¶

TrueWatch's scene custom explorer added MySQL database explorer template, can help you build MySQL log explorer with one click. In TrueWatch workspace 'Scene' - 'Explorer' - 'Built-in Explorer Templates', click 'MySQL Explorer Template', directly create MySQL log explorer. If already collected related logs, can view and analyze data via that log explorer.

Integration Template Updates¶

New Host System EthTool Integration Document and View¶

EthTool metrics include network interface in/out traffic, in/out packets, dropped packets, etc.

New Host System Conntrack Integration Document and View¶

Conntrack performance metrics include successful search entries count, inserted packets count, connection count, etc.

February 22, 2022¶

Added Log Configure Pipeline Script¶

Pipeline used for log data parsing, cutting logs of various formats into structured data meeting our requirements by defining parsing rules. TrueWatch provides three log Pipeline text processing methods:

DataKit: After installing DataKit on server, configure DataKit's log collector and corresponding pipeline file in terminal tool, process text data.
DCA: DataKit Control APP, DataKit's desktop client application, need install first, after installation can view and edit DataKit default built-in log pipeline files and custom manually added pipeline files in client.
Pipelines: Supports manually configuring and viewing log pipeline files in TrueWatch workspace, no need login DataKit server to operate.

Added IFrame Chart Component¶

TrueWatch added IFrame chart component, supports you configuring https or http link address. In IFrame URL can directly input external address to view, or use template variables in IFrame URL to view. For more configuration details, refer to document IFrame.

Added Event Detail History Records, Associated SLO¶

Observation optimized basic attributes, status & trend, and associated events layout on event detail page, and added history records and associated SLO. Click event name in abnormal event list to view.

Added event history records, supports viewing detection object host, abnormal/recovery time, and duration.

If configured SLO in monitor, then can view associated SLO, including SLO name, compliance rate, remaining quota, target, etc.

Added Save Snapshot Default Enable Absolute Time¶

TrueWatch added default enable absolute time when saving snapshot.

If choose enable absolute time when saving snapshot, after share displays absolute time when snapshot saved. E.g., when saving snapshot, choose last 15 minutes, you open snapshot link at 14:00, displays previous absolute time data.
If choose disable absolute time when saving snapshot, after share displays absolute time when snapshot saved. E.g., when saving snapshot, choose last 15 minutes, you open snapshot link at 14:00, displays 13:45 ~ 14:00 data.

For more snapshot share details, refer to document Snapshot.

Optimized Monitor Data Gap Trigger Event Configuration and Trigger Condition Unit Prompt¶

TrueWatch added three data gap status configurations 'Trigger Data Gap Event', 'Trigger Recovery Event', 'Do Not Trigger Event'.

Metric data monitor configuration, need manually configure data gap handling strategy.
Log data monitor configuration, by default selects 'Trigger Recovery Event' strategy, no need data gap configuration, obtains 'Normal' condition period as data gap recovery event period.

Optimized Chart Query Expression Calculation Unit¶

TrueWatch optimized chart query expression calculation unit logic. If query A has unit, query A operation result with number also has unit. E.g.: A's unit is KB, then A+100 unit also KB. For more details, refer to document Chart Query.

Added 'Time Series' Pay-As-You-Go Mode¶

TrueWatch added 'Time Series' pay-as-you-go mode, and optimized 'DataKit+Time Series' pay-as-you-go mode. Specific billing mode refer to document Pay-As-You-Go.

Other Optimized Functions¶

Chart query data source log, APM, security check, and network supports select all (*).
Chart query text, button style, and text prompt optimization.
Workspace operation button iconized, e.g., edit, delete, etc.
Other UI display optimization.

January 20, 2022¶

Added Open API and API Key Management¶

'TrueWatch' supports obtaining and updating TrueWatch workspace data via calling Open API interface method. Before calling API interface, need create API Key as authentication method. For more details, refer to document API Key Management.

Added Metric Dictionary, Metric Unit Management¶

After metric data collected, can view all collected measurements and their metrics and tags in TrueWatch workspace's 'Metric Dictionary', supports manually setting unit for custom metric data. For more details, refer to document Metric Dictionary.

Metrics help you understand system overall availability, e.g., server CPU usage, website load time, etc. Combined with TrueWatch provided logs and trace, helps you quickly locate and resolve faults.
Tags help you correlate data. TrueWatch supports unifying all metrics, logs, trace data reporting to workspace. By tagging collected data with same tags for correlated query, can help you perform correlation analysis, discover and resolve potential risks.

Added Scene Chart Funnel Chart¶

Funnel chart generally suitable for process analysis with standardization, long cycle, many links. Comparing data of each link via funnel chart can intuitively contrast problems. Also funnel chart suitable for website business process analysis, displaying user final conversion rate from entering website to achieving purchase, and each step conversion rate. For more details, refer to document Funnel Chart.

Added Scene Dashboard Save to Built-in Views, and Bind Label Data¶

After dashboard view created, can click 'Settings' button, select 'Save to Built-in Views', save dashboard view to built-in views' 'User Views'.

When saving dashboard view to built-in views, supports selecting binding relationship, select binding relationship 'label'. After saved to built-in views, can view saved dashboard view in TrueWatch workspace 'Management' - 'Built-in Views' 'User Views'. Also because set binding relationship label:*, in infrastructure hosts, containers detail pages that set 'Label Attribute' can view bound built-in views. For more details, refer to document Save Dashboard as Built-in View.

Added Container Detail Page Associated Pod¶

On container detail page, supports you viewing related Pod (association field: pod_name) basic information and within selected time component range performance metric status via detail page. For more container association query, refer to document Container.
Note: Viewing related Pod in container detail requires matching field 'pod_name', otherwise cannot view related Pod page in container detail.

Added Monitor Group Management¶

TrueWatch added grouping function supports you custom creating meaningful monitor combinations when setting monitors, supports filtering corresponding monitors via 'Grouping', convenient for grouping management of various monitors.

Note: - Each monitor when created must select a group, by default selects 'Default Group'. - When a group deleted, monitors under deleted group automatically categorized under 'Default Group'.

Added Log Explorer, Table Chart, Log Stream Chart Formatting Configuration¶

TrueWatch added formatting configuration allows you hide sensitive log data content or highlight log data content need view, also can quickly filter by replacing original log content. Supports formatting configuration in log explorer, table chart, log stream chart.

Optimized Mute Management, Added Disable/Enable Rules¶

TrueWatch added mute rule disable/enable function, helps you quickly disable/enable mute tasks. For more details, refer to document Mute Management.

Enable: Mute rule executes according to normal process.
Disable: Mute rule not effective. If set mute notification strategy, selected is before start 'xx minutes' and mute notification operation not yet executed, notification not executed.

Note: Enable/disable rules both generate operation audit events, can view in TrueWatch workspace 'Management' - 'Basic Settings' operation audit.

Added Log Pipeline User Manual¶

TrueWatch added log pipeline user manual, helps you understand how to assist writing Pipeline scripts via DataKit built-in debugging tools.

# Debug script example
datakit --pl datakit.p --txt '2022-01-12T18:40:51.962+0800 WARN diskio diskio/input.go:320 Error gathering disk info: open /run/udev/data/b252:1: no such file or directory'

# Extraction success example
Extracted data(drop: false, cost: 3.108038ms):
{
  "code": "diskio/input.go:320",
  "level": "WARN",
  "message": "2022-01-12T18:40:51.962+0800 WARN diskio diskio/input.go:320 Error gathering disk info: open /run/udev/data/b252:1: no such file or directory",
  "module": "diskio",
  "msg": "Error gathering disk info: open /run/udev/data/b252:1: no such file or directory",
  "time": 1641984051962000000
}

Added DQL Outer Functions¶

TrueWatch added two outer functions rate() and irate().

rate(): Calculates average change rate of a metric over certain time range. Suitable for alerts and slowly moving counters.
irate(): Calculates instantaneous change rate of a metric over certain time range, suitable for drawing volatile, fast-changing counters.

For more details, refer to document DQL Outer Functions.

Changelog (2022)¶

December 29, 2022¶

TrueWatch Updates¶

New MFA Authentication Management¶

New Workspace IP Access Login Whitelist¶

New Service Network Topology¶

Other Feature Optimizations¶

DataKit Updates¶

Intelligent Inspection Updates¶

Official Intelligent Inspections Discontinued¶

New Cloud Account Instance Dimension Bill Inspection¶

New Alibaba Cloud Preemptible Instance Survival Inspection¶

December 15, 2022¶

TrueWatch Updates¶

New Profile and Network Billing Items¶

TrueWatch Registration Process Optimization¶

New Warroom Feature¶

New Dashboard Carousel Function¶

Workspace Binding External SLS Index¶

Trace Explorer Chart Display Optimization¶

Other Feature Optimizations¶

DataKit Updates¶

Intelligent Inspection Updates¶

Intelligent Inspection Adds Guide Page¶

Intelligent Inspection Components Support Configuring Custom Redirect Links¶

Best Practices Updates¶

December 1, 2022¶

TrueWatch Updates¶

APM Adds Service List Feature¶

Dashboard Chart Usage Experience Optimization¶

View Variable Usage Experience Optimization¶

RUM Session Explorer Adjustment¶

Events Add Mobile App Redirect Option¶

Other Feature Optimizations¶

DataKit Updates¶

Intelligent Inspection Updates¶

Best Practices Updates¶

November 17, 2022¶

TrueWatch Updates¶

Pipeline Usage Experience Optimization¶

Event Optimization¶

Custom Explorer Supports Selecting More Data Types¶

Member Management Adds Member Grouping Function¶

Optimized Built-in View Binding Function¶

New One-Click Import/Export of Dashboards, Custom Explorers, Monitors within Workspace¶

APM Log Page Adds Custom Association Fields¶

Other Feature Optimizations¶

DataKit Updates¶

Best Practices Updates¶

November 11, 2022¶

TrueWatch Adds 'China Region 4 (Guangzhou)' Site¶

November 3, 2022¶

TrueWatch Updates¶

New TrueWatch and SLS Joint Solution¶

Optimized New User Guide Page¶

Added 3 Intelligent Inspection Configuration Documents¶

New Trace Error Tracking Explorer¶

Optimized Time Series Chart, Overview Chart Period-over-Period Comparison Function¶

Other Feature Optimizations¶

DataKit Updates¶

Best Practices Updates¶

October 20, 2022¶

TrueWatch Updates¶

Monitoring Optimization¶

Added Outlier Detection¶

Mutation Detection Logic Optimization¶

Range Detection Logic Optimization¶

Other Optimizations¶

Scene Chart Optimization¶

Added Histogram Chart Component¶

Chart Function Optimization¶

Explorer Optimization¶

Explorer Supports Analysis Mode¶

Optimized Associated Log Viewing Experience¶

Log Index Optimization¶

Optimized Metric Analysis Chart Query¶

Other Feature Optimizations¶

DataKit Updates¶

Best Practices Updates¶

September 29, 2022¶