Tokenization Rules¶

Under the Doris storage engine, the queryString function performs matching based on full-text indexing, and its semantics are directly related to the tokenization results. The TrueWatch underlying tokenizer follows the Default Word Boundaries specification of Unicode Standard Annex #29, splitting text into independent Tokens for indexing and querying.

Word Boundary Determination Logic¶

UAX #29 determines whether a word boundary occurs between two adjacent characters through a set of rules (WB1–WB999). The core rules are as follows:

WB5: No boundary between letters (ALetter), e.g., hello is considered one token.
WB6 / WB7: Letter + specific punctuation (MidLetter, MidNumLet) + letter, no boundary, e.g., example.com, handlepongmessage:devops are considered one token.
WB8 / WB9 / WB10: No boundary between digits, or between letters and digits, e.g., 32 and 3 in 32.3 are considered a numeric sequence (. as MidNumLet does not break the numeric boundary).
WB13a / WB13b: No boundary between underscore (ExtendNumLet) and letters/digits, e.g., user_name is considered one token.
WB999: For all other cases not covered by the above rules, a word boundary is produced.

Delimiters (Produce Word Boundaries)¶

The following characters are considered delimiters by the tokenizer, causing word boundaries before and after them, and will be split into separate tokens during querying:

Character	Unicode Property	Description	Example (Original Text → Tokenization Result)
`␣` (Space)	`White_Space` / `WSegSpace`	Basic delimiter	`hello world` → `hello`, `world`
`/`	`Other`	URL path separator	`http://example.com/path` → `http`, `example.com`, `path`
`?`	`Other`	URL query parameter start marker	`?query=1` → `query`, `1`
`=`	`Other`	Key-value pair separator	`key=value` → `key`, `value`
`&`	`Other`	URL parameter connector	`a=1&b=2` → `a`, `1`, `b`, `2`
`@`	`Other`	Email separator	`user@example.com` → `user`, `example.com`

Note

If the above characters need to be matched literally in a queryString query, it is recommended to use escaping or quotation marks. However, since they are removed during the tokenization stage, even if wrapped in quotes, the underlying index still matches based on the tokenized tokens.

Token Components (Do Not Produce Word Boundaries)¶

The following characters are considered part of a token by the tokenizer and do not produce word boundaries before or after them:

Character	Unicode Property	Description	Example
`.` (Dot)	`MidNumLet`	Part of domain names, identifiers	`example.com` is considered one token
`_` (Underscore)	`ExtendNumLet`	Part of identifiers	`user_name` is considered one token
`:` (Colon)	`MidLetter`	Part of namespaces, identifiers	`handlepongmessage:devops` is considered one token

Tokenization Examples¶

Example 1: URL Tokenization¶

Original text:

https://api.example.com/v1/users?id=123&name=test

Tokenization result (only valid tokens retained):

https, api.example.com, v1, users, id, 123, name, test

Example 2: Email Address Tokenization¶

Original text:

admin@example.com

Tokenization result:

admin, example.com

Example 3: Underscore Identifier¶

Original text:

user_name = "Alice"

Tokenization result:

user_name, Alice

Example 4: Colon-Separated Identifier¶

Original text:

handlepongmessage:devops

Tokenization result:

handlepongmessage:devops

Impact on Queries¶

Understanding the tokenization rules helps write more precise queryString queries:

Query Statement	Actual Matching Logic	Description
`queryString("example.com")`	Matches documents containing the token `example.com`	Dot is part of the token
`queryString("user_name")`	Matches documents containing the token `user_name`	Underscore is part of the token
`queryString("user@example")`	Matches documents containing `user` OR `example`	`@` is a delimiter, the query is split into two tokens, connected by `OR` by default
`queryString("id=123")`	Matches documents containing `id` OR `123`	`=` is a delimiter
`queryString("a/b")`	Matches documents containing `a` OR `b`	`/` is a delimiter

Notes¶

Limitations of Phrase Queries¶

Double quotes are used for exact phrase matching, requiring tokens to appear adjacently in the index. However, since delimiters (such as @, /, =, etc.) are removed during the tokenization stage, phrase queries cannot distinguish specific delimiters in the original text. For example:

queryString("user@example.com")

Actually matches documents where the tokens user and example.com appear adjacently, which could match texts like user@example.com and user example.com. If strict matching of the original string containing @ is required, consider using the match or regexp function instead.

Chinese Tokenization¶

For Chinese, Japanese, and other CJK characters, the default UAX #29 rule splits by character (each character is considered an independent token), consistent with the "Chinese: split by character" behavior of the search function.

Case Sensitivity¶

queryString matching is case-insensitive by default.