> Discover all available pages from the documentation index: https://mastra.zisheng.pro/llms.txt

# Tool 调用准确性 Scorer

Mastra 提供两种 Tool Call Accuracy Scorer，用于评估 LLM 是否从可用选项中选择了正确的 Tool：

1. **基于代码的 Scorer**——使用精确 Tool 匹配进行确定性评估
2. **基于 LLM 的 Scorer**——使用 AI 评估恰当性的语义评估

## 如何选择 Scorer

### 以下情况使用基于代码的 Scorer：

- 需要**确定且可复现**的结果
- 希望测试**精确 Tool 匹配**
- 需要验证**特定的 Tool 序列**
- 优先考虑速度和成本（不调用 LLM）
- 正在运行自动化测试

### 以下情况使用基于 LLM 的 Scorer：

- 需要从**语义层面理解**恰当性
- Tool 选择取决于**上下文和意图**
- 希望处理澄清请求等**边缘情况**
- 需要对评分决策作出**解释**
- 正在评估**生产环境中的 Agent 行为**

## 基于代码的 Tool Call Accuracy Scorer

`@mastra/evals/scorers/prebuilt` 中的 `createToolCallAccuracyScorerCode()` 函数基于精确 Tool 匹配提供确定性的二元评分，并支持严格和宽松两种评估模式以及 Tool 调用顺序验证。

### 参数

**expectedTool** (`string`): 针对给定任务应调用的 Tool 名称。提供 expectedToolOrder 时忽略此项。

**strictMode** (`boolean`): 控制评估的严格程度。对于单 Tool 模式：仅接受精确的单次 Tool 调用。对于顺序检查模式：Tool 必须完全匹配，且不允许额外 Tool。

**expectedToolOrder** (`string[]`): 按预期调用顺序排列的 Tool 名称数组。提供后会启用顺序检查模式，并忽略 expectedTool 参数。

此函数返回 MastraScorer 类的实例。有关 `.run()` 方法及其输入/输出的详情，请参阅 [MastraScorer 参考](https://mastra.zisheng.pro/reference/evals/mastra-scorer)。

### 评估模式

基于代码的 Scorer 以两种不同模式运行：

#### 单 Tool 模式

未提供 `expectedToolOrder` 时，Scorer 会评估单个 Tool 的选择：

- **标准模式（strictMode: false）**：只要调用了预期 Tool 即返回 `1`，无论是否调用其他 Tool
- **严格模式（strictMode: true）**：仅当恰好调用一个 Tool 且与预期 Tool 匹配时返回 `1`

#### 顺序检查模式

提供 `expectedToolOrder` 时，Scorer 会验证 Tool 调用顺序：

- **严格顺序（strictMode: true）**：必须严格按指定顺序调用 Tool，且不得调用额外 Tool
- **灵活顺序（strictMode: false）**：预期 Tool 必须以正确的相对顺序出现（允许额外 Tool）

## 基于代码的评分详情

- **二元分数**：始终返回 0 或 1
- **确定性**：相同输入始终产生相同输出
- **快速**：无需外部 API 调用

### 基于代码的 Scorer 选项

```typescript
// Standard mode - passes if expected tool is called
const lenientScorer = createCodeScorer({
  expectedTool: 'search-tool',
  strictMode: false,
})

// Strict mode - only passes if exactly one tool is called
const strictScorer = createCodeScorer({
  expectedTool: 'search-tool',
  strictMode: true,
})

// Order checking with strict mode
const strictOrderScorer = createCodeScorer({
  expectedTool: 'step1-tool',
  expectedToolOrder: ['step1-tool', 'step2-tool', 'step3-tool'],
  strictMode: true, // no extra tools allowed
})
```

### 基于代码的 Scorer 结果

```typescript
{
  runId: string,
  preprocessStepResult: {
    expectedTool: string,
    actualTools: string[],
    strictMode: boolean,
    expectedToolOrder?: string[],
    hasToolCalls: boolean,
    correctToolCalled: boolean,
    correctOrderCalled: boolean | null,
    toolCallInfos: ToolCallInfo[]
  },
  score: number // Always 0 or 1
}
```

## 基于代码的 Scorer 示例

基于代码的 Scorer 根据精确 Tool 匹配提供确定性的二元评分（0 或 1）。

### 正确选择 Tool

```typescript
const scorer = createToolCallAccuracyScorerCode({
  expectedTool: 'weather-tool',
})

// Simulate LLM input and output with tool call
const inputMessages = [
  createTestMessage({
    content: 'What is the weather like in New York today?',
    role: 'user',
    id: 'input-1',
  }),
]

const output = [
  createTestMessage({
    content: 'Let me check the weather for you.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-123',
        toolName: 'weather-tool',
        args: { location: 'New York' },
        result: { temperature: '72°F', condition: 'sunny' },
        state: 'result',
      }),
    ],
  }),
]

const run = createAgentTestRun({ inputMessages, output })
const result = await scorer.run(run)

console.log(result.score) // 1
console.log(result.preprocessStepResult?.correctToolCalled) // true
```

### 严格模式评估

仅当恰好调用一个 Tool 时才通过：

```typescript
const strictScorer = createToolCallAccuracyScorerCode({
  expectedTool: 'weather-tool',
  strictMode: true,
})

// Multiple tools called - fails in strict mode
const output = [
  createTestMessage({
    content: 'Let me help you with that.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-1',
        toolName: 'search-tool',
        args: {},
        result: {},
        state: 'result',
      }),
      createToolInvocation({
        toolCallId: 'call-2',
        toolName: 'weather-tool',
        args: { location: 'New York' },
        result: { temperature: '20°C' },
        state: 'result',
      }),
    ],
  }),
]

const result = await strictScorer.run(run)
console.log(result.score) // 0 - fails because multiple tools were called
```

### Tool 顺序验证

验证 Tool 是否按特定顺序调用：

```typescript
const orderScorer = createToolCallAccuracyScorerCode({
  expectedTool: 'auth-tool', // ignored when order is specified
  expectedToolOrder: ['auth-tool', 'fetch-tool'],
  strictMode: true, // no extra tools allowed
})

const output = [
  createTestMessage({
    content: 'I will authenticate and fetch the data.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-1',
        toolName: 'auth-tool',
        args: { token: 'abc123' },
        result: { authenticated: true },
        state: 'result',
      }),
      createToolInvocation({
        toolCallId: 'call-2',
        toolName: 'fetch-tool',
        args: { endpoint: '/data' },
        result: { data: ['item1'] },
        state: 'result',
      }),
    ],
  }),
]

const result = await orderScorer.run(run)
console.log(result.score) // 1 - correct order
```

### 灵活顺序模式

只要预期 Tool 保持相对顺序，就允许调用额外 Tool：

```typescript
const flexibleOrderScorer = createToolCallAccuracyScorerCode({
  expectedTool: 'auth-tool',
  expectedToolOrder: ['auth-tool', 'fetch-tool'],
  strictMode: false, // allows extra tools
})

const output = [
  createTestMessage({
    content: 'Performing comprehensive operation.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-1',
        toolName: 'auth-tool',
        args: { token: 'abc123' },
        result: { authenticated: true },
        state: 'result',
      }),
      createToolInvocation({
        toolCallId: 'call-2',
        toolName: 'log-tool', // Extra tool - OK in flexible mode
        args: { message: 'Starting fetch' },
        result: { logged: true },
        state: 'result',
      }),
      createToolInvocation({
        toolCallId: 'call-3',
        toolName: 'fetch-tool',
        args: { endpoint: '/data' },
        result: { data: ['item1'] },
        state: 'result',
      }),
    ],
  }),
]

const result = await flexibleOrderScorer.run(run)
console.log(result.score) // 1 - auth-tool comes before fetch-tool
```

## 基于 LLM 的 Tool Call Accuracy Scorer

`@mastra/evals/scorers/prebuilt` 中的 `createToolCallAccuracyScorerLLM()` 函数使用 LLM 评估 Agent 调用的 Tool 是否适合用户请求，提供语义评估而非精确匹配。

### 参数

**model** (`MastraModelConfig`): 用于评估 Tool 恰当性的 LLM 模型

**availableTools** (`Array<{name: string, description: string}>`): 可用 Tool 及其描述的列表，用于提供上下文

### 特性

基于 LLM 的 Scorer 提供以下特性：

- **语义评估**：理解上下文和用户意图
- **恰当性评估**：区分“有帮助”和“恰当”的 Tool
- **澄清处理**：识别 Agent 何时恰当地请求澄清
- **缺失 Tool 检测**：识别本应调用但未调用的 Tool
- **生成推理**：解释评分决策

### 评估流程

1. **提取 Tool 调用**：识别 Agent 输出中提及的 Tool
2. **分析恰当性**：根据用户请求评估每个 Tool
3. **生成分数**：根据恰当 Tool 调用数与总 Tool 调用数计算分数
4. **生成推理**：提供易于理解的说明

## 基于 LLM 的评分详情

- **小数分数**：返回 0.0 到 1.0 之间的值
- **上下文感知**：考虑用户意图和恰当性
- **可解释**：提供评分理由

### 基于 LLM 的 Scorer 选项

```typescript
// Basic configuration
const basicLLMScorer = createLLMScorer({
  model: 'openai/gpt-5.6-sol',
  availableTools: [
    { name: 'tool1', description: 'Description 1' },
    { name: 'tool2', description: 'Description 2' }
  ]
});

// With different model
const customModelScorer = createLLMScorer({
  model: 'openai/gpt-5', // More powerful model for complex evaluations
  availableTools: [...]
});
```

### 基于 LLM 的 Scorer 结果

```typescript
{
  runId: string,
  score: number,  // 0.0 to 1.0
  reason: string, // Human-readable explanation
  analyzeStepResult: {
    evaluations: Array<{
      toolCalled: string,
      wasAppropriate: boolean,
      reasoning: string
    }>,
    missingTools?: string[]
  }
}
```

## 基于 LLM 的 Scorer 示例

基于 LLM 的 Scorer 使用 AI 评估 Tool 选择是否适合用户请求。

### 基本 LLM 评估

```typescript
const llmScorer = createToolCallAccuracyScorerLLM({
  model: 'openai/gpt-5.6-sol',
  availableTools: [
    {
      name: 'weather-tool',
      description: 'Get current weather information for any location',
    },
    {
      name: 'calendar-tool',
      description: 'Check calendar events and scheduling',
    },
    {
      name: 'search-tool',
      description: 'Search the web for general information',
    },
  ],
})

const inputMessages = [
  createTestMessage({
    content: 'What is the weather like in San Francisco today?',
    role: 'user',
    id: 'input-1',
  }),
]

const output = [
  createTestMessage({
    content: 'Let me check the current weather for you.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-123',
        toolName: 'weather-tool',
        args: { location: 'San Francisco', date: 'today' },
        result: { temperature: '68°F', condition: 'foggy' },
        state: 'result',
      }),
    ],
  }),
]

const run = createAgentTestRun({ inputMessages, output })
const result = await llmScorer.run(run)

console.log(result.score) // 1.0 - appropriate tool usage
console.log(result.reason) // "The agent correctly used the weather-tool to address the user's request for weather information."
```

### 处理不恰当的 Tool 使用

```typescript
const inputMessages = [
  createTestMessage({
    content: 'What is the weather in Tokyo?',
    role: 'user',
    id: 'input-1',
  }),
]

const inappropriateOutput = [
  createTestMessage({
    content: 'Let me search for that information.',
    role: 'assistant',
    id: 'output-1',
    toolInvocations: [
      createToolInvocation({
        toolCallId: 'call-456',
        toolName: 'search-tool', // Less appropriate than weather-tool
        args: { query: 'Tokyo weather' },
        result: { results: ['Tokyo weather data...'] },
        state: 'result',
      }),
    ],
  }),
]

const run = createAgentTestRun({ inputMessages, output: inappropriateOutput })
const result = await llmScorer.run(run)

console.log(result.score) // 0.5 - partially appropriate
console.log(result.reason) // "The agent used search-tool when weather-tool would have been more appropriate for a direct weather query."
```

### 评估澄清请求

LLM Scorer 能识别 Agent 何时恰当地请求澄清：

```typescript
const vagueInput = [
  createTestMessage({
    content: 'I need help with something',
    role: 'user',
    id: 'input-1'
  })
];

const clarificationOutput = [
  createTestMessage({
    content: 'I'd be happy to help! Could you please provide more details about what you need assistance with?',
    role: 'assistant',
    id: 'output-1',
    // No tools called - asking for clarification instead
  })
];

const run = createAgentTestRun({
  inputMessages: vagueInput,
  output: clarificationOutput
});
const result = await llmScorer.run(run);

console.log(result.score); // 1.0 - appropriate to ask for clarification
console.log(result.reason); // "The agent appropriately asked for clarification rather than calling tools with insufficient information."
```

## 比较两种 Scorer

以下示例对同一数据使用两种 Scorer：

```typescript
import {
  createToolCallAccuracyScorerCode as createCodeScorer,
  createToolCallAccuracyScorerLLM as createLLMScorer,
} from '@mastra/evals/scorers/prebuilt'

// Setup both scorers
const codeScorer = createCodeScorer({
  expectedTool: 'weather-tool',
  strictMode: false,
})

const llmScorer = createLLMScorer({
  model: 'openai/gpt-5.6-sol',
  availableTools: [
    { name: 'weather-tool', description: 'Get weather information' },
    { name: 'search-tool', description: 'Search the web' },
  ],
})

// Test data
const run = createAgentTestRun({
  inputMessages: [
    createTestMessage({
      content: 'What is the weather?',
      role: 'user',
      id: 'input-1',
    }),
  ],
  output: [
    createTestMessage({
      content: 'Let me find that information.',
      role: 'assistant',
      id: 'output-1',
      toolInvocations: [
        createToolInvocation({
          toolCallId: 'call-1',
          toolName: 'search-tool',
          args: { query: 'weather' },
          result: { results: ['weather data'] },
          state: 'result',
        }),
      ],
    }),
  ],
})

// Run both scorers
const codeResult = await codeScorer.run(run)
const llmResult = await llmScorer.run(run)

console.log('Code Scorer:', codeResult.score) // 0 - wrong tool
console.log('LLM Scorer:', llmResult.score) // 0.3 - partially appropriate
console.log('LLM Reason:', llmResult.reason) // Explains why search-tool is less appropriate
```

## 相关内容

- [Answer Relevancy Scorer](https://mastra.zisheng.pro/reference/evals/answer-relevancy)
- [Completeness Scorer](https://mastra.zisheng.pro/reference/evals/completeness)
- [Faithfulness Scorer](https://mastra.zisheng.pro/reference/evals/faithfulness)
- [Custom Scorers](https://mastra.zisheng.pro/docs/evals/custom-scorers)