Prompt 对齐 Scorer

createPromptAlignmentScorerLLM() 函数创建一个 Scorer，从意图理解、要求满足情况、响应完整性和格式恰当性等方面评估 Agent 响应与用户 prompt 的对齐程度。

参数

model:

MastraModelConfig

用于评估 prompt 与响应对齐程度的语言模型

options:

PromptAlignmentOptions

Scorer 的配置选项

.run() 返回值

score:

number

介于 0 和 scale 之间的多维对齐分数（默认为 0-1）

reason:

string

易于理解的 Prompt Alignment 评估说明及详细分项结果

.run() 返回以下结构的结果：

{
  runId: string,
  score: number,
  reason: string,
  analyzeStepResult: {
    intentAlignment: {
      score: number,
      primaryIntent: string,
      isAddressed: boolean,
      reasoning: string
    },
    requirementsFulfillment: {
      requirements: Array<{
        requirement: string,
        isFulfilled: boolean,
        reasoning: string
      }>,
      overallScore: number
    },
    completeness: {
      score: number,
      missingElements: string[],
      reasoning: string
    },
    responseAppropriateness: {
      score: number,
      formatAlignment: boolean,
      toneAlignment: boolean,
      reasoning: string
    },
    overallAssessment: string
  }
}

评分详情

Scorer 配置

你可以调整 scale 参数和评估模式来自定义 Prompt Alignment Scorer，以满足评分需求。

const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    scale: 10, // Score from 0-10 instead of 0-1
    evaluationMode: 'both', // 'user', 'system', or 'both' (default)
  },
})

多维分析

Prompt Alignment 从四个关键维度评估响应，并采用根据评估模式调整的加权评分：

User 模式（'user'）

仅评估与用户 prompt 的对齐情况：

1. 意图对齐（权重 40%）：响应是否处理用户的核心请求
2. 要求满足情况（权重 30%）：是否满足所有用户要求
3. 完整性（权重 20%）：响应是否针对用户需求提供了充分细节
4. 响应恰当性（权重 10%）：格式和语气是否符合用户预期

System 模式（'system'）

仅评估对 system 指南的遵循情况：

1. 意图对齐（权重 35%）：响应是否遵循 system 行为指南
2. 要求满足情况（权重 35%）：是否遵守所有 system 约束
3. 完整性（权重 15%）：响应是否遵循所有 system 规则
4. 响应恰当性（权重 15%）：格式和语气是否符合 system 规范

Both 模式（'both'——默认）

结合评估用户与 system 的对齐情况：

• 用户对齐：占最终分数的 70%（使用 user 模式权重）
• System 合规性：占最终分数的 30%（使用 system 模式权重）
• 兼顾评估用户满意度和 system 遵循情况

评分公式

**User 模式：

Weighted Score = (intent_score × 0.4) + (requirements_score × 0.3) +
                 (completeness_score × 0.2) + (appropriateness_score × 0.1)
Final Score = Weighted Score × scale

**System 模式：

Weighted Score = (intent_score × 0.35) + (requirements_score × 0.35) +
                 (completeness_score × 0.15) + (appropriateness_score × 0.15)
Final Score = Weighted Score × scale

**Both 模式（默认）：

User Score = (user dimensions with user weights)
System Score = (system dimensions with system weights)
Weighted Score = (User Score × 0.7) + (System Score × 0.3)
Final Score = Weighted Score × scale

权重分配依据：

• User 模式：优先考虑意图（40%）和要求（30%），以提高用户满意度
• System 模式：对行为合规性（35%）和约束（35%）给予同等权重
• Both 模式：采用 70/30 的分配，确保以用户需求为主，同时保持 system 合规性

分数解读

• 0.9-1.0 = 所有维度的对齐均非常出色
• 0.8-0.9 = 对齐情况很好，仅有少量不足
• 0.7-0.8 = 对齐情况良好，但缺少部分要求或完整性
• 0.6-0.7 = 对齐情况一般，存在明显不足
• 0.4-0.6 = 对齐情况较差，存在严重问题
• 0.0-0.4 = 对齐情况很差，响应未能有效处理 prompt

各模式的适用场景

User 模式（'user'）——适用于：

• 从用户满意度角度评估客服响应
• 从用户视角测试内容生成质量
• 衡量响应处理用户问题的程度
• 纯粹关注请求满足情况，不考虑 system 约束

System 模式（'system'）——适用于：

• 审核 AI 安全性以及对行为指南的遵循情况
• 确保 Agent 遵循品牌措辞和语气要求
• 验证对内容政策和约束的遵循情况
• 测试 system 层面的行为一致性

Both 模式（'both'）——适用于（默认、推荐）：

• 全面评估 AI Agent 的表现
• 兼顾用户满意度和 system 合规性
• 同时重视用户要求和 system 要求的生产监控
• 整体评估 prompt 与响应的对齐情况

常见使用场景

代码生成评估

非常适合评估：

• 编程任务完成情况
• 代码质量和完整性
• 对编码要求的遵循情况
• 格式规范（函数、类等）

// Example: API endpoint creation
const codePrompt = 'Create a REST API endpoint with authentication and rate limiting'
// Scorer evaluates: intent (API creation), requirements (auth + rate limiting),
// completeness (full implementation), format (code structure)

指令遵循评估

非常适合：

• 验证任务完成情况
• 多步骤指令遵循情况
• 要求合规性检查
• 教育内容评估

// Example: Multi-requirement task
const taskPrompt =
  'Write a Python class with initialization, validation, error handling, and documentation'
// Scorer tracks each requirement individually and provides detailed breakdown

内容格式验证

适用于：

• 格式规范合规性
• 样式指南遵循情况
• 输出结构验证
• 响应恰当性检查

// Example: Structured output
const formatPrompt =
  'Explain the differences between let and const in JavaScript using bullet points'
// Scorer evaluates content accuracy AND format compliance

Agent 响应质量

衡量 AI Agent 遵循用户指令的程度：

const agent = new Agent({
  id: 'coding-assistant',
  name: 'CodingAssistant',
  instructions: 'You are a helpful coding assistant. Always provide working code examples.',
  model: 'openai/gpt-5.6-sol',
})

// Evaluate comprehensive alignment (default)
const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: { evaluationMode: 'both' }, // Evaluates both user intent and system guidelines
})

// Evaluate just user satisfaction
const userScorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: { evaluationMode: 'user' }, // Focus only on user request fulfillment
})

// Evaluate system compliance
const systemScorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: { evaluationMode: 'system' }, // Check adherence to system instructions
})

const result = await scorer.run(agentRun)

Prompt Engineering 优化

测试不同 prompt 以改进对齐情况：

const prompts = [
  'Write a function to calculate factorial',
  'Create a Python function that calculates factorial with error handling for negative inputs',
  'Implement a factorial calculator in Python with: input validation, error handling, and docstring',
]

// Compare alignment scores to find the best prompt
for (const prompt of prompts) {
  const result = await scorer.run(createTestRun(prompt, response))
  console.log(`Prompt alignment: ${result.score}`)
}

Multi-Agent 系统评估

比较不同 Agent 或模型：

const agents = [agent1, agent2, agent3];
const testPrompts = [...]; // Array of test prompts

for (const agent of agents) {
  let totalScore = 0;
  for (const prompt of testPrompts) {
    const response = await agent.run(prompt);
    const evaluation = await scorer.run({ input: prompt, output: response });
    totalScore += evaluation.score;
  }
  console.log(`${agent.name} average alignment: ${totalScore / testPrompts.length}`);
}

示例

基本配置

import { createPromptAlignmentScorerLLM } from '@mastra/evals'

const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
})

// Evaluate a code generation task
const result = await scorer.run({
  input: [
    {
      role: 'user',
      content: 'Write a Python function to calculate factorial with error handling',
    },
  ],
  output: {
    role: 'assistant',
    text: `def factorial(n):
    if n < 0:
        raise ValueError("Factorial not defined for negative numbers")
    if n == 0:
        return 1
    return n * factorial(n-1)`,
  },
})
// Result: { score: 0.95, reason: "Excellent alignment - function addresses intent, includes error handling..." }

自定义配置示例

// Configure scale and evaluation mode
const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    scale: 10, // Score from 0-10 instead of 0-1
    evaluationMode: 'both', // 'user', 'system', or 'both' (default)
  },
})

// User-only evaluation - focus on user satisfaction
const userScorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: { evaluationMode: 'user' },
})

// System-only evaluation - focus on compliance
const systemScorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: { evaluationMode: 'system' },
})

const result = await scorer.run(testRun)
// Result: { score: 8.5, reason: "Score: 8.5 out of 10 - Good alignment with both user intent and system guidelines..." }

特定格式评估

// Evaluate bullet point formatting
const result = await scorer.run({
  input: [
    {
      role: 'user',
      content: 'List the benefits of TypeScript in bullet points',
    },
  ],
  output: {
    role: 'assistant',
    text: 'TypeScript provides static typing, better IDE support, and enhanced code reliability.',
  },
})
// Result: Lower appropriateness score due to format mismatch (paragraph vs bullet points)

出色的对齐示例

在此示例中，响应完全处理了用户的 prompt，并满足所有要求。

src/example-excellent-prompt-alignment.ts

import { createPromptAlignmentScorerLLM } from '@mastra/evals/scorers/prebuilt'

const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
})

const inputMessages = [
  {
    role: 'user',
    content:
      'Write a Python function to calculate factorial with error handling for negative numbers',
  },
]

const outputMessage = {
  text: `def factorial(n):
    """Calculate factorial of a number."""
    if n < 0:
        raise ValueError("Factorial not defined for negative numbers")
    if n == 0 or n == 1:
        return 1
    return n * factorial(n - 1)`,
}

const result = await scorer.run({
  input: inputMessages,
  output: outputMessage,
})

console.log(result)

出色的对齐输出

该输出获得高分，因为它完全符合意图、满足所有要求，并使用了恰当的格式。

{
  score: 0.95,
  reason: 'The score is 0.95 because the response perfectly addresses the primary intent of creating a factorial function and fulfills all requirements including Python implementation, error handling for negative numbers, and proper documentation. The code format is appropriate and the implementation is complete.'
}

部分对齐示例

在此示例中，响应处理了核心意图，但遗漏了一些要求或存在格式问题。

src/example-partial-prompt-alignment.ts

import { createPromptAlignmentScorerLLM } from '@mastra/evals/scorers/prebuilt'

const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
})

const inputMessages = [
  {
    role: 'user',
    content: 'List the benefits of TypeScript in bullet points',
  },
]

const outputMessage = {
  text: 'TypeScript provides static typing, better IDE support, and enhanced code reliability through compile-time error checking.',
}

const result = await scorer.run({
  input: inputMessages,
  output: outputMessage,
})

console.log(result)

部分对齐输出

该输出的内容虽然准确，但没有遵循要求的格式（项目符号列表），因此分数较低。

{
  score: 0.75,
  reason: 'The score is 0.75 because the response addresses the intent of explaining TypeScript benefits and provides accurate information, but fails to use the requested bullet point format, resulting in lower appropriateness scoring.'
}

对齐较差的示例

在此示例中，响应未能满足用户的具体要求。

src/example-poor-prompt-alignment.ts

import { createPromptAlignmentScorerLLM } from '@mastra/evals/scorers/prebuilt'

const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
})

const inputMessages = [
  {
    role: 'user',
    content:
      'Write a Python class with initialization, validation, error handling, and documentation',
  },
]

const outputMessage = {
  text: `class Example:
    def __init__(self, value):
        self.value = value`,
}

const result = await scorer.run({
  input: inputMessages,
  output: outputMessage,
})

console.log(result)

对齐较差的输出

该输出仅满足部分要求，缺少验证、错误处理和文档，因此分数较低。

{
  score: 0.35,
  reason: 'The score is 0.35 because while the response addresses the basic intent of creating a Python class with initialization, it fails to include validation, error handling, and documentation as specifically requested, resulting in incomplete requirement fulfillment.'
}

评估模式示例

User 模式——仅关注用户 prompt

评估响应处理用户请求的程度，忽略 system 指令：

src/example-user-mode.ts

const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: { evaluationMode: 'user' },
})

const result = await scorer.run({
  input: {
    inputMessages: [
      {
        role: 'user',
        content: 'Explain recursion with an example',
      },
    ],
    systemMessages: [
      {
        role: 'system',
        content: 'Always provide code examples in Python',
      },
    ],
  },
  output: {
    text: 'Recursion is when a function calls itself. For example: factorial(5) = 5 * factorial(4)',
  },
})
// Scores high for addressing user request, even without Python code

System 模式——仅关注 system 指南

评估对 system 行为指南和约束的遵循情况：

src/example-system-mode.ts

const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: { evaluationMode: 'system' },
})

const result = await scorer.run({
  input: {
    systemMessages: [
      {
        role: 'system',
        content: 'You are a helpful assistant. Always be polite, concise, and provide examples.',
      },
    ],
    inputMessages: [
      {
        role: 'user',
        content: 'What is machine learning?',
      },
    ],
  },
  output: {
    text: 'Machine learning is a subset of AI where computers learn from data. For example, spam filters learn to identify unwanted emails by analyzing patterns in previously marked spam.',
  },
})
// Evaluates politeness, conciseness, and example provision

Both 模式——组合评估（默认）

通过加权评分同时评估用户意图满足情况和 system 合规性（用户占 70%，system 占 30%）：

src/example-both-mode.ts

const scorer = createPromptAlignmentScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: { evaluationMode: 'both' }, // This is the default
})

const result = await scorer.run({
  input: {
    systemMessages: [
      {
        role: 'system',
        content: 'Always provide code examples when explaining programming concepts',
      },
    ],
    inputMessages: [
      {
        role: 'user',
        content: 'Explain how to reverse a string',
      },
    ],
  },
  output: {
    text: `To reverse a string, you can iterate through it backwards. Here's an example in Python:
    
    def reverse_string(s):
        return s[::-1]
    
    # Usage: reverse_string("hello") returns "olleh"`,
  },
})
// High score for both addressing the user's request AND following system guidelines

与其他 Scorer 的比较

方面	Prompt Alignment	Answer Relevancy	Faithfulness
重点	多维 prompt 遵循情况	查询与响应的相关性	以上下文为依据的程度
评估内容	意图、要求、完整性、格式	与查询的语义相似度	与上下文的事实一致性
使用场景	常规 prompt 遵循	信息检索	基于 RAG/上下文的系统
维度	4 个加权维度	单一相关性维度	单一 Faithfulness 维度

相关内容

• Answer Relevancy Scorer: 评估查询与响应的相关性
• Faithfulness Scorer: 衡量以上下文为依据的程度
• Tool Call Accuracy Scorer: 评估 Tool 选择
• Custom Scorers: 创建自己的评估指标