> Discover all available pages from the documentation index: https://mastra.zisheng.pro/zh-HK/llms.txt

# Tool 呼叫準確度評分器

Mastra 提供兩種 Tool 呼叫準確度評分器，用於評估 LLM 有否從可用選項中選取正確的 Tool：

1. **程式碼式評分器** - 使用精確 Tool 配對進行確定性評估
2. **LLM 式評分器** - 使用 AI 評估合適程度的語意評估

## 選擇評分器

### 適合使用程式碼式評分器的情況

- 你需要**確定且可重現**的結果
- 你想測試**精確 Tool 配對**
- 你需要驗證**特定 Tool 次序**
- 速度及成本是首要考慮（無需呼叫 LLM）
- 你正在執行自動化測試

### 適合使用 LLM 式評分器的情況

- 你需要從**語意層面理解**合適程度
- Tool 的選擇取決於**語境及意圖**
- 你想處理要求澄清等**邊緣情況**
- 你需要評分決定的**解釋**
- 你正在評估正式環境中的 **Agent 行為**

## 程式碼式 Tool 呼叫準確度評分器

`@mastra/evals/scorers/prebuilt` 的 `createToolCallAccuracyScorerCode()` 函式會根據精確 Tool 配對提供確定性的二元評分，並支援嚴格及寬鬆評估模式，以及 Tool 呼叫次序驗證。

### 參數

**expectedTool** (`string`): 針對指定工作應呼叫的 Tool 名稱。提供 expectedToolOrder 時會忽略此項。

**strictMode** (`boolean`): 控制評估的嚴格程度。單一 Tool 模式只接受完全相符的單一 Tool 呼叫；次序檢查模式則要求 Tool 完全相符，不允許額外 Tool。

**expectedToolOrder** (`string[]`): 按預期呼叫次序排列的 Tool 名稱陣列。提供後會啟用次序檢查模式，並忽略 expectedTool 參數。

此函式會傳回 MastraScorer 類別的實例。有關 `.run()` 方法及其輸入／輸出的詳情，請參閱 [MastraScorer 參考](https://mastra.zisheng.pro/zh-HK/reference/evals/mastra-scorer)。

### 評估模式

程式碼式評分器有兩種不同的運作模式：

#### 單一 Tool 模式

未有提供 `expectedToolOrder` 時，評分器會評估單一 Tool 選擇：

- **標準模式（strictMode: false）**：只要呼叫了預期 Tool，便傳回 `1`，不論有否呼叫其他 Tool
- **嚴格模式（strictMode: true）**：只有在剛好呼叫一個 Tool，而該 Tool 與預期 Tool 相符時，才傳回 `1`

#### 次序檢查模式

提供 `expectedToolOrder` 時，評分器會驗證 Tool 呼叫次序：

- **嚴格次序（strictMode: true）**：必須完全按指定次序呼叫 Tool，且不可有額外 Tool
- **彈性次序（strictMode: false）**：預期的 Tool 必須按正確的相對次序出現（允許額外 Tool）

## 程式碼式評分詳情

- **二元分數**：必定傳回 0 或 1
- **確定性**：相同輸入必定產生相同輸出
- **快速**：無需呼叫外部 API

### 程式碼式評分器選項

```typescript
// Standard mode - passes if expected tool is called
const lenientScorer = createCodeScorer({
  expectedTool: 'search-tool',
  strictMode: false,
})

// Strict mode - only passes if exactly one tool is called
const strictScorer = createCodeScorer({
  expectedTool: 'search-tool',
  strictMode: true,
})

// Order checking with strict mode
const strictOrderScorer = createCodeScorer({
  expectedTool: 'step1-tool',
  expectedToolOrder: ['step1-tool', 'step2-tool', 'step3-tool'],
  strictMode: true, // no extra tools allowed
})
```

### 程式碼式評分器結果

```typescript
{
  runId: string,
  preprocessStepResult: {
    expectedTool: string,
    actualTools: string[],
    strictMode: boolean,
    expectedToolOrder?: string[],
    hasToolCalls: boolean,
    correctToolCalled: boolean,
    correctOrderCalled: boolean | null,
    toolCallInfos: ToolCallInfo[]
  },
  score: number // Always 0 or 1
}
```

## 程式碼式評分器範例

程式碼式評分器會根據精確 Tool 配對，提供確定性的二元評分（0 或 1）。

### 選擇正確的 Tool

```typescript
const scorer = createToolCallAccuracyScorerCode({
  expectedTool: 'weather-tool',
})

// Simulate LLM input and output with tool call
const inputMessages = [
  createTestMessage({
    content: 'What is the weather like in New York today?',
    role: 'user',
    id: 'input-1',
  }),
]

const output = [
  createTestMessage({
    content: 'Let me check the weather for you.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-123',
        toolName: 'weather-tool',
        args: { location: 'New York' },
        result: { temperature: '72°F', condition: 'sunny' },
        state: 'result',
      }),
    ],
  }),
]

const run = createAgentTestRun({ inputMessages, output })
const result = await scorer.run(run)

console.log(result.score) // 1
console.log(result.preprocessStepResult?.correctToolCalled) // true
```

### 嚴格模式評估

只有剛好呼叫一個 Tool 時才會通過：

```typescript
const strictScorer = createToolCallAccuracyScorerCode({
  expectedTool: 'weather-tool',
  strictMode: true,
})

// Multiple tools called - fails in strict mode
const output = [
  createTestMessage({
    content: 'Let me help you with that.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-1',
        toolName: 'search-tool',
        args: {},
        result: {},
        state: 'result',
      }),
      createToolInvocation({
        toolCallId: 'call-2',
        toolName: 'weather-tool',
        args: { location: 'New York' },
        result: { temperature: '20°C' },
        state: 'result',
      }),
    ],
  }),
]

const result = await strictScorer.run(run)
console.log(result.score) // 0 - fails because multiple tools were called
```

### Tool 次序驗證

驗證 Tool 有否按特定次序呼叫：

```typescript
const orderScorer = createToolCallAccuracyScorerCode({
  expectedTool: 'auth-tool', // ignored when order is specified
  expectedToolOrder: ['auth-tool', 'fetch-tool'],
  strictMode: true, // no extra tools allowed
})

const output = [
  createTestMessage({
    content: 'I will authenticate and fetch the data.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-1',
        toolName: 'auth-tool',
        args: { token: 'abc123' },
        result: { authenticated: true },
        state: 'result',
      }),
      createToolInvocation({
        toolCallId: 'call-2',
        toolName: 'fetch-tool',
        args: { endpoint: '/data' },
        result: { data: ['item1'] },
        state: 'result',
      }),
    ],
  }),
]

const result = await orderScorer.run(run)
console.log(result.score) // 1 - correct order
```

### 彈性次序模式

只要預期 Tool 保持相對次序，便允許額外 Tool：

```typescript
const flexibleOrderScorer = createToolCallAccuracyScorerCode({
  expectedTool: 'auth-tool',
  expectedToolOrder: ['auth-tool', 'fetch-tool'],
  strictMode: false, // allows extra tools
})

const output = [
  createTestMessage({
    content: 'Performing comprehensive operation.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-1',
        toolName: 'auth-tool',
        args: { token: 'abc123' },
        result: { authenticated: true },
        state: 'result',
      }),
      createToolInvocation({
        toolCallId: 'call-2',
        toolName: 'log-tool', // Extra tool - OK in flexible mode
        args: { message: 'Starting fetch' },
        result: { logged: true },
        state: 'result',
      }),
      createToolInvocation({
        toolCallId: 'call-3',
        toolName: 'fetch-tool',
        args: { endpoint: '/data' },
        result: { data: ['item1'] },
        state: 'result',
      }),
    ],
  }),
]

const result = await flexibleOrderScorer.run(run)
console.log(result.score) // 1 - auth-tool comes before fetch-tool
```

## LLM 式 Tool 呼叫準確度評分器

`@mastra/evals/scorers/prebuilt` 的 `createToolCallAccuracyScorerLLM()` 函式會使用 LLM，評估 Agent 所呼叫的 Tool 是否適合用戶要求，提供語意評估而非精確配對。

### 參數

**model** (`MastraModelConfig`): 用於評估 Tool 合適程度的 LLM 模型

**availableTools** (`Array<{name: string, description: string}>`): 可用 Tool 及其說明的清單，用作評估語境

### 功能

LLM 式評分器提供以下功能：

- **語意評估**：理解語境及用戶意圖
- **合適程度評估**：區分「有幫助」及「合適」的 Tool
- **澄清處理**：識別 Agent 在何時適當地要求澄清
- **遺漏 Tool 偵測**：識別本應呼叫的 Tool
- **推理生成**：提供評分決定的解釋

### 評估流程

1. **擷取 Tool 呼叫**：識別 Agent 輸出中提及的 Tool
2. **分析合適程度**：根據用戶要求評估每個 Tool
3. **產生分數**：按合適 Tool 呼叫佔全部 Tool 呼叫的比例計算分數
4. **產生推理**：提供容易理解的解釋

## LLM 式評分詳情

- **小數分數**：傳回介乎 0.0 至 1.0 的值
- **理解語境**：考慮用戶意圖及合適程度
- **具解釋性**：提供分數的推理

### LLM 式評分器選項

```typescript
// Basic configuration
const basicLLMScorer = createLLMScorer({
  model: 'openai/gpt-5.6-sol',
  availableTools: [
    { name: 'tool1', description: 'Description 1' },
    { name: 'tool2', description: 'Description 2' }
  ]
});

// With different model
const customModelScorer = createLLMScorer({
  model: 'openai/gpt-5', // More powerful model for complex evaluations
  availableTools: [...]
});
```

### LLM 式評分器結果

```typescript
{
  runId: string,
  score: number,  // 0.0 to 1.0
  reason: string, // Human-readable explanation
  analyzeStepResult: {
    evaluations: Array<{
      toolCalled: string,
      wasAppropriate: boolean,
      reasoning: string
    }>,
    missingTools?: string[]
  }
}
```

## LLM 式評分器範例

LLM 式評分器會使用 AI 評估所選 Tool 是否適合用戶要求。

### 基本 LLM 評估

```typescript
const llmScorer = createToolCallAccuracyScorerLLM({
  model: 'openai/gpt-5.6-sol',
  availableTools: [
    {
      name: 'weather-tool',
      description: 'Get current weather information for any location',
    },
    {
      name: 'calendar-tool',
      description: 'Check calendar events and scheduling',
    },
    {
      name: 'search-tool',
      description: 'Search the web for general information',
    },
  ],
})

const inputMessages = [
  createTestMessage({
    content: 'What is the weather like in San Francisco today?',
    role: 'user',
    id: 'input-1',
  }),
]

const output = [
  createTestMessage({
    content: 'Let me check the current weather for you.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-123',
        toolName: 'weather-tool',
        args: { location: 'San Francisco', date: 'today' },
        result: { temperature: '68°F', condition: 'foggy' },
        state: 'result',
      }),
    ],
  }),
]

const run = createAgentTestRun({ inputMessages, output })
const result = await llmScorer.run(run)

console.log(result.score) // 1.0 - appropriate tool usage
console.log(result.reason) // "The agent correctly used the weather-tool to address the user's request for weather information."
```

### 處理不合適的 Tool 使用情況

```typescript
const inputMessages = [
  createTestMessage({
    content: 'What is the weather in Tokyo?',
    role: 'user',
    id: 'input-1',
  }),
]

const inappropriateOutput = [
  createTestMessage({
    content: 'Let me search for that information.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-456',
        toolName: 'search-tool', // Less appropriate than weather-tool
        args: { query: 'Tokyo weather' },
        result: { results: ['Tokyo weather data...'] },
        state: 'result',
      }),
    ],
  }),
]

const run = createAgentTestRun({ inputMessages, output: inappropriateOutput })
const result = await llmScorer.run(run)

console.log(result.score) // 0.5 - partially appropriate
console.log(result.reason) // "The agent used search-tool when weather-tool would have been more appropriate for a direct weather query."
```

### 評估澄清要求

LLM 評分器可識別 Agent 有否在適當情況下要求澄清：

```typescript
const vagueInput = [
  createTestMessage({
    content: 'I need help with something',
    role: 'user',
    id: 'input-1'
  })
];

const clarificationOutput = [
  createTestMessage({
    content: 'I'd be happy to help! Could you please provide more details about what you need assistance with?',
    role: 'assistant',
    id: 'output-1',
    // No tools called - asking for clarification instead
  })
];

const run = createAgentTestRun({
  inputMessages: vagueInput,
  output: clarificationOutput
});
const result = await llmScorer.run(run);

console.log(result.score); // 1.0 - appropriate to ask for clarification
console.log(result.reason); // "The agent appropriately asked for clarification rather than calling tools with insufficient information."
```

## 比較兩種評分器

以下範例在同一組資料上使用兩種評分器：

```typescript
import {
  createToolCallAccuracyScorerCode as createCodeScorer,
  createToolCallAccuracyScorerLLM as createLLMScorer,
} from '@mastra/evals/scorers/prebuilt'

// Setup both scorers
const codeScorer = createCodeScorer({
  expectedTool: 'weather-tool',
  strictMode: false,
})

const llmScorer = createLLMScorer({
  model: 'openai/gpt-5.6-sol',
  availableTools: [
    { name: 'weather-tool', description: 'Get weather information' },
    { name: 'search-tool', description: 'Search the web' },
  ],
})

// Test data
const run = createAgentTestRun({
  inputMessages: [
    createTestMessage({
      content: 'What is the weather?',
      role: 'user',
      id: 'input-1',
    }),
  ],
  output: [
    createTestMessage({
      content: 'Let me find that information.',
      role: 'assistant',
      id: 'output-1',
      toolInvocations: [
        createToolInvocation({
          toolCallId: 'call-1',
          toolName: 'search-tool',
          args: { query: 'weather' },
          result: { results: ['weather data'] },
          state: 'result',
        }),
      ],
    }),
  ],
})

// Run both scorers
const codeResult = await codeScorer.run(run)
const llmResult = await llmScorer.run(run)

console.log('Code Scorer:', codeResult.score) // 0 - wrong tool
console.log('LLM Scorer:', llmResult.score) // 0.3 - partially appropriate
console.log('LLM Reason:', llmResult.reason) // Explains why search-tool is less appropriate
```

## 相關內容

- [答案相關性評分器](https://mastra.zisheng.pro/zh-HK/reference/evals/answer-relevancy)
- [完整度評分器](https://mastra.zisheng.pro/zh-HK/reference/evals/completeness)
- [忠實度評分器](https://mastra.zisheng.pro/zh-HK/reference/evals/faithfulness)
- [自訂評分器](https://mastra.zisheng.pro/zh-HK/docs/evals/custom-scorers)