Output Quality Evaluation¶

Output Quality Evaluation checks whether LLM outputs meet expectations. By configuring evaluation rules, you can run evaluations on production model calls or data in a specified scope, and view scores and rule hit results.

Currently, two evaluation types are supported: Prompt Eval and Schema Eval.

Create Evaluator¶

Go to Agent Monitoring > Evaluation and click Create Evaluator.

Configure the following information:

1. Select the evaluation type. When using Prompt Eval, you can select an evaluator Skill template or describe the evaluation goal to generate the configuration;
2. Select the evaluation target type;
3. Confirm the evaluation instructions, input structure, and result type;
4. Use test data to perform a test run;
5. Publish the evaluator after confirming that the results meet expectations.

Prompt Eval¶

Prompt Eval uses prompts to judge model inputs or outputs, and is suitable for evaluation scenarios such as content quality, relevance, completeness, and security.

During configuration, you need to set:

• The evaluation prompt;
• The evaluation result and scoring rules.

It is recommended to clearly specify the evaluation goal, judgment criteria, and return format in the prompt to ensure stable evaluation results.

Describe Evaluation Goal¶

After selecting Prompt Eval, you can enter natural language requirements in "Describe Evaluation Goal" to generate an evaluation configuration. The description should clearly specify the evaluation object, judgment criteria, exceptions, and the expected result type. For example:

Check whether the LLM reply is polite. Only evaluate the manner of expression, not factual accuracy. Return whether it passed and explain the reason.

After generation, review the target type, evaluation instructions, result type, and recommended input, and modify them as needed. In particular, check whether the exception conditions are accurate and whether the selected data contains the evidence required for judgment. Descriptions without a valid evaluation dimension need to be supplemented and regenerated.

You can go back to the previous step to modify the description. When selecting a platform template, if the page prompts you to clear the existing description, confirm before switching. Description generation applies to Prompt Eval; Schema Eval follows its corresponding structure validation configuration.

Custom Evaluation Input¶

In "Evaluation Input Structure" under "Evaluation Configuration", you can edit the input template passed to the evaluator, combining descriptive text with Trace or Span context fields. The evaluation instructions define how to judge, and the input template determines which data is used as the basis for judgment.

The template references variables with double curly braces, and supports field paths, array indexes, and [*]. For example, when evaluating the root node's input and output, you can use:

User input:
{{root.input}}

Final output:
{{root.output}}

Field paths should use variables available from the current evaluation perspective. The template must contain valid variables; plain text without variables cannot proceed to the next step. For evaluations involving tool calls, confirm that the template includes the tool name, parameters, return results, or execution status needed for judgment.

In the test run results, expand "Actual Evaluation Input" to check whether the content after variable substitution meets expectations. If a field is not collected, first check the data collection and input configuration; do not judge data completeness based on scores alone.

Manually entered fixed content is used only as the input for this evaluation and does not modify the original Trace. After modifying the evaluation instructions or other key configurations, you need to test run again.

Schema Eval¶

Schema Eval is used to check whether model outputs conform to the specified data structure, and is suitable for scenarios that require the model to return JSON or other structured content.

During configuration, you need to set the target Schema. After the evaluation is executed, the system checks whether the fields, types, and structure of the output content meet the requirements.

Test Evaluator¶

Before publishing an evaluator, select a time range and real Trace samples to perform a test run. When using a custom input template, also verify the actual evaluation input, results, and reasons for judgment.

Test results display the evaluation output, score, or structure validation result. You can adjust the prompt, Schema, or judgment rules based on the results and test again.

You can proceed to the publishing step only after the current configuration has successfully completed a test run. Modifying key configurations such as evaluation instructions invalidates previous test run results, and you need to test run again. After confirming that the results meet expectations, click Publish to make the evaluator available for formal evaluation tasks.

Run Evaluation¶

Online Evaluation¶

Online evaluation is used to continuously check model call data generated in the production environment.

Select the target application and evaluator, and configure the data scope to be evaluated. After the task is enabled, the system evaluates the data that meets the conditions according to the configuration.

Batch Evaluation¶

Batch evaluation is used to evaluate historical model call data in a specified time range or under query conditions.

When creating a batch evaluation task:

1. Select the target application;
2. Set the time range and filter conditions;
3. Select the evaluator to use;
4. Confirm and start the evaluation task.

View Evaluation Results¶

On the evaluation results page, you can view:

• The evaluator and evaluation method;
• The evaluation status;
• Score or rule validation results;
• Model input and output;
• The evaluation time;
• Failure reason.

You can filter evaluation results by conditions such as application, evaluator, evaluation status, and time range.

Manual Annotation¶

With manual annotation, you can review evaluation results in combination with model input, output, and call chain, and add samples to the dataset.

Create Annotation Queue¶

Create a queue of items to be annotated based on Trace query conditions. Data such as marked failures and Agent judgment anomalies is supported.

The queue shows the number of pending, completed, and skipped items, as well as the current processing progress. The queue uses a shared pending annotation list; assignment, claiming, and arbitration workflows for owners are not currently supported.

Submit Annotation¶

In the annotation workbench, view the input, output, call chain, tool calls, user feedback, and existing evaluation results, and then fill in:

After filling in, you can choose "Save and Next" to continue annotating, or "Save and Add to Dataset" to accumulate samples.

Re-annotating retains the historical records. Manual annotation results are labeled with their source and displayed separately from platform evaluation results and external Scores.

Dataset¶

Datasets are formed from production Trace conditions and manual annotation conditions, and are used to centrally manage samples and run batch evaluations.

Start Batch Evaluation¶

Select samples in the dataset and start a batch evaluation. When the task starts, the sample scope for this run is fixed; subsequent changes to dataset members do not affect the running task.

Permissions¶

Configuration operations such as evaluator management, evaluation execution, annotation queue and dataset management, and manual annotation all use the Agent Monitoring > Application Configuration Management permission. This permission applies to both Agent and LLM integrations. Viewing related Trace data also requires the corresponding data query permission. Datasets, annotations, and evaluation results all follow the access permissions of the current workspace and the original Trace.