> Discover all available pages from the documentation index: https://mastra.zisheng.pro/zh-HK/llms.txt

# 自訂評分器

Mastra 提供統一的 `createScorer` 工廠，讓你在每個步驟使用 JavaScript 函數或以 LLM 為基礎的提示物件，建立自訂評估邏輯。這種靈活性讓你可為評估管線的每個部分選擇最合適的方法。

## 四步管線

Mastra 中所有評分器都遵循一致的四步評估管線：

1. **preprocess**（選用）：準備或轉換輸入／輸出資料
2. **analyze**（選用）：執行評估分析並收集分析結果
3. **generateScore**（必須）：將分析轉換為數值分數
4. **generateReason**（選用）：產生方便閱讀的說明

每個步驟都可以使用**函數**或**提示物件**（以 LLM 為基礎的評估），讓你可按需要結合確定性演算法與 AI 判斷。

## 函數與提示物件的比較

**函數**使用 JavaScript 實作確定性邏輯，適合：

- 準則清晰的演算法評估
- 對效能要求嚴格的情境
- 與現有依賴套件整合
- 一致且可重現的結果

**提示物件**使用 LLM 作為評估裁判，適合：

- 需要類似人類判斷的主觀評估
- 難以用演算法編寫的複雜準則
- 自然語言理解工作
- 細緻的情境評估

**「提示物件」的意思：** 該步驟不使用函數，而是使用包含 `description` + `createPrompt`（以及 `outputSchema`，供 `preprocess`／`analyze` 使用）的物件。這個物件指示 Mastra 為該步驟執行裁判 LLM，並將結構化輸出儲存在 `results.<step>StepResult`。

你可以在單一評分器內混合使用不同方法，例如以函數預處理資料，再以 LLM 分析質素。

## 初始化評分器

每個評分器都由 `createScorer` 工廠函數開始。此函數需要 id 和描述，也可選擇性接受類型規格及裁判設定。

```typescript
import { createScorer } from '@mastra/core/evals';

const glutenCheckerScorer = createScorer({
  id: 'gluten-checker',
  description: 'Check if recipes contain gluten ingredients',
  judge: {                    // Optional: for prompt object steps
    model: 'openai/gpt-5.6-sol',
    instructions: 'You are a Chef that identifies if recipes contain gluten.'
  }
})
// Chain step methods here
.preprocess(...)
.analyze(...)
.generateScore(...)
.generateReason(...)
```

只有當你打算在任何步驟使用提示物件時，才需要裁判設定。個別步驟可以用本身的裁判設定覆寫這項預設設定。

如果所有步驟都以函數為基礎，系統便不會呼叫裁判，也不會有裁判輸出。如要查看 LLM 輸出，請將至少一個步驟定義為提示物件，並讀取相應的步驟結果（例如 `results.analyzeStepResult`）。

### 最精簡裁判範例（提示物件）

此範例在 `analyze` 中使用提示物件，因此系統會執行裁判，而其結構化輸出可從 `results.analyzeStepResult` 取得。

```typescript
import { createScorer } from '@mastra/core/evals'
import { z } from 'zod'

const quoteSourcesScorer = createScorer({
  id: 'quote-sources',
  description: 'Check if the response includes sources',
  judge: {
    model: 'openai/gpt-5-mini',
    instructions: 'You are a strict evaluator.',
  },
})
  .analyze({
    description: 'Detect whether sources are present',
    outputSchema: z.object({
      hasSources: z.boolean(),
      sources: z.array(z.string()),
    }),
    createPrompt: ({ run }) => `
Does the response contain sources? Extract them as a list.

Response:
${run.output}
`,
  })
  .generateScore(({ results }) => (results.analyzeStepResult.hasSources ? 1 : 0))

// Run the scorer and inspect judge output
const result = await quoteSourcesScorer.run({
  input: 'What is the capital of France?',
  output: 'Paris is the capital of France [1]. Source: [1] Wikipedia',
})

console.log(result.score) // 1
console.log(result.analyzeStepResult) // { hasSources: true, sources: ["Wikipedia"] }
```

### 用於 Agent 評估的 Agent 類型

為確保類型安全，並同時兼容即時 Agent 評分及 Trace 評分，建立用於 Agent 評估的評分器時，請使用 `type: 'agent'`。這讓你可用同一個評分器為 Agent 評分，也可用它為 Trace 評分：

```typescript
const myScorer = createScorer({
  type: 'agent', // Automatically handles agent input/output types
}).generateScore(({ run, results }) => {
  // run.output is automatically typed as ScorerRunOutputForAgent
  // run.input is automatically typed as ScorerRunInputForAgent
})
```

## 步驟詳解

### preprocess 步驟（選用）

當你需要擷取特定元素、篩選內容或轉換複雜資料結構時，此步驟會準備輸入／輸出資料。

**函數：** `({ run, results }) => any`

```typescript
const glutenCheckerScorer = createScorer(...)
.preprocess(({ run }) => {
  // Extract and clean recipe text
  const recipeText = run.output.text.toLowerCase();
  const wordCount = recipeText.split(' ').length;

  return {
    recipeText,
    wordCount,
    hasCommonGlutenWords: /flour|wheat|bread|pasta/.test(recipeText)
  };
})
```

**提示物件：** 使用 `description`、`outputSchema` 和 `createPrompt`，為以 LLM 為基礎的預處理建立結構。

```typescript
const glutenCheckerScorer = createScorer(...)
.preprocess({
  description: 'Extract ingredients from the recipe',
  outputSchema: z.object({
    ingredients: z.array(z.string()),
    cookingMethods: z.array(z.string())
  }),
  createPrompt: ({ run }) => `
    Extract all ingredients and cooking methods from this recipe:
    ${run.output.text}

    Return JSON with ingredients and cookingMethods arrays.
  `
})
```

**資料流程：** 結果可供後續步驟透過 `results.preprocessStepResult` 使用

### analyze 步驟（選用）

執行核心評估分析，收集用於評分決策的分析結果。

**函數：** `({ run, results }) => any`

```typescript
const glutenCheckerScorer = createScorer({...})
.preprocess(...)
.analyze(({ run, results }) => {
  const { recipeText, hasCommonGlutenWords } = results.preprocessStepResult;

  // Simple gluten detection algorithm
  const glutenKeywords = ['wheat', 'flour', 'barley', 'rye', 'bread'];
  const foundGlutenWords = glutenKeywords.filter(word =>
    recipeText.includes(word)
  );

  return {
    isGlutenFree: foundGlutenWords.length === 0,
    detectedGlutenSources: foundGlutenWords,
    confidence: hasCommonGlutenWords ? 0.9 : 0.7
  };
})
```

**提示物件：** 使用 `description`、`outputSchema` 和 `createPrompt` 進行以 LLM 為基礎的分析。

```typescript
const glutenCheckerScorer = createScorer({...})
.preprocess(...)
.analyze({
  description: 'Analyze recipe for gluten content',
  outputSchema: z.object({
    isGlutenFree: z.boolean(),
    glutenSources: z.array(z.string()),
    confidence: z.number().min(0).max(1)
  }),
  createPrompt: ({ run, results }) => `
    Analyze this recipe for gluten content:
    "${results.preprocessStepResult.recipeText}"

    Look for wheat, barley, rye, and hidden sources like soy sauce.
    Return JSON with isGlutenFree, glutenSources array, and confidence (0-1).
  `
})
```

**資料流程：** 結果可供後續步驟透過 `results.analyzeStepResult` 使用

### `generateScore` 步驟（必須）

將分析結果轉換為數值分數。這是管線中唯一的必要步驟。

**函數：** `({ run, results }) => number`

```typescript
const glutenCheckerScorer = createScorer({...})
.preprocess(...)
.analyze(...)
.generateScore(({ results }) => {
  const { isGlutenFree, confidence } = results.analyzeStepResult;

  // Return 1 for gluten-free, 0 for contains gluten
  // Weight by confidence level
  return isGlutenFree ? confidence : 0;
})
```

**提示物件：** 請參閱 [`createScorer`](https://mastra.zisheng.pro/zh-HK/reference/evals/create-scorer) API 參考，了解如何配合 generateScore 使用提示物件，包括必要的 `calculateScore` 函數。

**資料流程：** 分數會以 `score` 參數形式提供給 generateReason

### `generateReason` 步驟（選用）

產生方便閱讀的分數說明，適用於除錯、提高透明度或提供用戶意見。

**函數：** `({ run, results, score }) => string`

```typescript
const glutenCheckerScorer = createScorer({...})
.preprocess(...)
.analyze(...)
.generateScore(...)
.generateReason(({ results, score }) => {
  const { isGlutenFree, glutenSources } = results.analyzeStepResult;

  if (isGlutenFree) {
    return `Score: ${score}. This recipe is gluten-free with no harmful ingredients detected.`;
  } else {
    return `Score: ${score}. Contains gluten from: ${glutenSources.join(', ')}`;
  }
})
```

**提示物件：** 使用 `description` 和 `createPrompt` 產生由 LLM 建立的說明。

```typescript
const glutenCheckerScorer = createScorer({...})
.preprocess(...)
.analyze(...)
.generateScore(...)
.generateReason({
  description: 'Explain the gluten assessment',
  createPrompt: ({ results, score }) => `
    Explain why this recipe received a score of ${score}.
    Analysis: ${JSON.stringify(results.analyzeStepResult)}

    Provide a clear explanation for someone with dietary restrictions.
  `
})
```

## 輸入篩選

Agent 對話可能包含數百則帶有 Tool 呼叫和資料部分的訊息，以及系統 metadata。大多數評分器只需要其中一部分資料。`prepareRun` 選項會在評分器管線執行前轉換執行資料，從而減少雜訊，讓評分器保持專注。

### 使用 `filterRun()` 進行宣告式篩選

[`filterRun()`](https://mastra.zisheng.pro/zh-HK/reference/evals/filter-run) 工具程式會根據宣告式選項建立 `prepareRun` 函數：

```typescript
import { createScorer, filterRun } from '@mastra/core/evals'

const toolScorer = createScorer({
  id: 'tool-quality',
  description: 'Evaluates tool usage quality',
  type: 'agent',
  prepareRun: filterRun({
    partTypes: ['tool-invocation', 'text'],
    maxRememberedMessages: 20,
  }),
}).generateScore(({ run }) => {
  // run.input.rememberedMessages has only tool and text messages, max 20
  return 1
})
```

常用選項包括：

- `partTypes`：只保留部分類型相符的訊息（例如 `'tool-invocation'`、`'text'`、`'reasoning'`）
- `toolNames`：只保留涉及指定 Tool 的訊息（例如 `['write_file', 'execute_command']`）
- `maxRememberedMessages`：限制情境視窗大小
- `dropRequestContext`、`dropGroundTruth`、`dropExpectedTrajectory`：移除未使用的欄位

如需完整選項清單，請參閱 [`filterRun()` 參考](https://mastra.zisheng.pro/zh-HK/reference/evals/filter-run)。

### 自訂 `prepareRun` 函數

如需實作 `filterRun()` 未涵蓋的邏輯，請直接編寫 `prepareRun` 函數：

```typescript
import { createScorer } from '@mastra/core/evals'

const customScorer = createScorer({
  id: 'recent-output',
  description: 'Scores only the last response',
  type: 'agent',
  prepareRun: (run) => ({
    ...run,
    output: run.output.slice(-1), // Keep only the last message
    requestContext: undefined,
  }),
})
.generateScore(({ run }) => {
  return run.output.length > 0 ? 1 : 0
})
```

`prepareRun` 函數亦可以是非同步函數。

> **系統訊息一律保留:** `filterRun()` 絕不會篩選 `systemMessages` 或 `taggedSystemMessages`。這些訊息包含 Agent 指示，是評分所需的重要情境。

## 範例：建立自訂評分器

Mastra 的自訂評分器使用 `createScorer`，並包含四個核心部分：

1. [**裁判設定**](#judge-configuration)
2. [**分析步驟**](#analysis-step)
3. [**產生分數**](#score-generation)
4. [**產生原因**](#reason-generation)

這些部分讓你可使用 LLM 作為裁判，定義自訂評估邏輯。如需完整 API 及設定選項，請參閱 [createScorer](https://mastra.zisheng.pro/zh-HK/reference/evals/create-scorer)。

```typescript
import { createScorer } from '@mastra/core/evals'
import { z } from 'zod'

export const GLUTEN_INSTRUCTIONS = `You are a Chef that identifies if recipes contain gluten.`

export const generateGlutenPrompt = ({
  output,
}: {
  output: string
}) => `Check if this recipe is gluten-free.

Check for:
- Wheat
- Barley
- Rye
- Common sources like flour, pasta, bread

Example with gluten:
"Mix flour and water to make dough"
Response: {
  "isGlutenFree": false,
  "glutenSources": ["flour"]
}

Example gluten-free:
"Mix rice, beans, and vegetables"
Response: {
  "isGlutenFree": true,
  "glutenSources": []
}

Recipe to analyze:
${output}

Return your response in this format:
{
  "isGlutenFree": boolean,
  "glutenSources": ["list ingredients containing gluten"]
}`

export const generateReasonPrompt = ({
  isGlutenFree,
  glutenSources,
}: {
  isGlutenFree: boolean
  glutenSources: string[]
}) => `Explain why this recipe is${isGlutenFree ? '' : ' not'} gluten-free.

${glutenSources.length > 0 ? `Sources of gluten: ${glutenSources.join(', ')}` : 'No gluten-containing ingredients found'}

Return your response in this format:
"This recipe is [gluten-free/contains gluten] because [explanation]"`

export const glutenCheckerScorer = createScorer({
  id: 'gluten-checker',
  description: 'Check if the output contains any gluten',
  judge: {
    model: 'openai/gpt-5-mini',
    instructions: GLUTEN_INSTRUCTIONS,
  },
})
  .analyze({
    description: 'Analyze the output for gluten',
    outputSchema: z.object({
      isGlutenFree: z.boolean(),
      glutenSources: z.array(z.string()),
    }),
    createPrompt: ({ run }) => {
      const { output } = run
      return generateGlutenPrompt({ output: output.text })
    },
  })
  .generateScore(({ results }) => {
    return results.analyzeStepResult.isGlutenFree ? 1 : 0
  })
  .generateReason({
    description: 'Generate a reason for the score',
    createPrompt: ({ results }) => {
      return generateReasonPrompt({
        glutenSources: results.analyzeStepResult.glutenSources,
        isGlutenFree: results.analyzeStepResult.isGlutenFree,
      })
    },
  })
```

### 裁判設定

設定 LLM 模型，並將其角色定義為領域專家。

```typescript
judge: {
  model: 'openai/gpt-5-mini',
  instructions: GLUTEN_INSTRUCTIONS,
}
```

### 分析步驟

定義 LLM 應如何分析輸入，以及要傳回哪種結構化輸出。

```typescript
.analyze({
  description: 'Analyze the output for gluten',
  outputSchema: z.object({
    isGlutenFree: z.boolean(),
    glutenSources: z.array(z.string()),
  }),
  createPrompt: ({ run }) => {
    const { output } = run;
    return generateGlutenPrompt({ output: output.text });
  },
})
```

分析步驟使用提示物件來：

- 清楚描述分析工作
- 使用 Standard JSON Schema 定義預期的輸出結構（包括布林結果及麩質來源清單）
- 根據輸入內容產生執行階段提示

### 產生分數

將 LLM 的結構化分析轉換為數值分數。

```typescript
.generateScore(({ results }) => {
  return results.analyzeStepResult.isGlutenFree ? 1 : 0;
})
```

分數產生函數會取得分析結果，並套用業務邏輯來產生分數。在此情況下，LLM 會直接判斷食譜是否不含麩質，因此我們使用該布林結果：不含麩質為 1，含麩質為 0。

### 產生原因

透過另一次 LLM 呼叫，為分數提供方便閱讀的說明。

```typescript
.generateReason({
  description: 'Generate a reason for the score',
  createPrompt: ({ results }) => {
    return generateReasonPrompt({
      glutenSources: results.analyzeStepResult.glutenSources,
      isGlutenFree: results.analyzeStepResult.isGlutenFree,
    });
  },
})
```

原因產生步驟會同時使用布林結果及分析步驟識別出的特定麩質來源，建立說明，協助用戶了解獲得該分數的原因。

## 完全不含麩質的範例

```typescript
const result = await glutenCheckerScorer.run({
  input: [{ role: 'user', content: 'Mix rice, beans, and vegetables' }],
  output: { text: 'Mix rice, beans, and vegetables' },
})

console.log('Score:', result.score)
console.log('Gluten sources:', result.analyzeStepResult.glutenSources)
console.log('Reason:', result.reason)
```

### 完全不含麩質的輸出

```typescript
{
  score: 1,
  analyzeStepResult: {
    isGlutenFree: true,
    glutenSources: []
  },
  reason: 'This recipe is gluten-free because rice, beans, and vegetables are naturally gluten-free ingredients that are safe for people with celiac disease.'
}
```

## 部分含麩質的範例

```typescript
const result = await glutenCheckerScorer.run({
  input: [{ role: 'user', content: 'Mix flour and water to make dough' }],
  output: { text: 'Mix flour and water to make dough' },
})

console.log('Score:', result.score)
console.log('Gluten sources:', result.analyzeStepResult.glutenSources)
console.log('Reason:', result.reason)
```

### 部分含麩質的輸出

```typescript
{
  score: 0,
  analyzeStepResult: {
    isGlutenFree: false,
    glutenSources: ['flour']
  },
  reason: 'This recipe is not gluten-free because it contains flour. Regular flour is made from wheat and contains gluten, making it unsafe for people with celiac disease or gluten sensitivity.'
}
```

## 不含麩質程度低的範例

```typescript
const result = await glutenCheckerScorer.run({
  input: [{ role: 'user', content: 'Add soy sauce and noodles' }],
  output: { text: 'Add soy sauce and noodles' },
})

console.log('Score:', result.score)
console.log('Gluten sources:', result.analyzeStepResult.glutenSources)
console.log('Reason:', result.reason)
```

### 不含麩質程度低的輸出

```typescript
{
  score: 0,
  analyzeStepResult: {
    isGlutenFree: false,
    glutenSources: ['soy sauce', 'noodles']
  },
  reason: 'This recipe is not gluten-free because it contains soy sauce, noodles. Regular soy sauce contains wheat and most noodles are made from wheat flour, both of which contain gluten and are unsafe for people with gluten sensitivity.'
}
```

**範例與資源：**

- [createScorer API 參考](https://mastra.zisheng.pro/zh-HK/reference/evals/create-scorer)：完整技術文件
- [內置評分器原始碼](https://github.com/mastra-ai/mastra/tree/main/packages/evals/src/scorers)：可供參考的實際實作