上下文相关性 Scorer

createContextRelevanceScorerLLM() 函数会创建一个 Scorer，用于评估所提供的上下文对于生成 Agent 响应的相关性和实用性。它使用加权的相关性等级，并对未使用的高相关性上下文和缺失信息进行扣分。

它尤其适用于以下用例：

内容生成评估

最适合在以下场景中评估上下文质量：

• 上下文使用情况很重要的 Chat 系统
• 需要详细评估相关性的 RAG pipeline
• 上下文缺失会影响质量的系统

上下文选择优化

适用于优化以下方面：

• 全面的上下文覆盖
• 有效利用上下文
• 识别上下文缺口

参数

model:

MastraModelConfig

用于评估上下文相关性的语言模型

options:

ContextRelevanceOptions

Scorer 的配置选项

注意：必须提供 context 或 contextExtractor。如果二者都提供，则 contextExtractor 优先。

.run() 返回值

score:

number

0 到 scale 之间的加权相关性分数（默认为 0-1）

reason:

string

对上下文相关性评估的易读说明

评分详情

加权相关性评分

Context Relevance 使用一种高级评分算法，考虑以下因素：

1. 相关性等级：每个上下文片段会被分类并赋予权重值：

   • high = 1.0（直接回应查询）
   • medium = 0.7（支持性信息）
   • low = 0.3（略微相关）
   • none = 0.0（完全不相关）

2. 使用情况检测：跟踪响应是否实际使用了相关上下文

3. 应用扣分（可通过 penalties 选项配置）：

   • 未使用的高相关性上下文：每个未使用的高相关性上下文扣除 unusedHighRelevanceContext（默认为 0.1）
   • 缺失上下文：对识别出的缺失信息最多扣除 maxMissingContextPenalty（默认为 0.5）

评分公式

Base Score = Σ(relevance_weights) / (num_contexts × 1.0)
Usage Penalty = count(unused_high_relevance) × unusedHighRelevanceContext
Missing Penalty = min(count(missing_context) × missingContextPerItem, maxMissingContextPenalty)

Final Score = max(0, Base Score - Usage Penalty - Missing Penalty) × scale

默认值：

• unusedHighRelevanceContext = 0.1（每个未使用的高相关性上下文扣 10%）
• missingContextPerItem = 0.15（每个缺失上下文项扣 15%）
• maxMissingContextPenalty = 0.5（上下文缺失最多扣 50%）
• scale = 1

分数解读

• 0.9-1.0：优秀——所有上下文都高度相关且已使用
• 0.7-0.8：良好——大部分相关，仅有少量缺口
• 0.4-0.6：一般——存在大量不相关或未使用的上下文
• 0.2-0.3：较差——大部分上下文不相关
• 0.0-0.1：非常差——未找到相关上下文

原因分析

reason 字段提供以下信息：

• 每个上下文片段的相关性等级（high/medium/low/none）
• 响应实际使用了哪些上下文
• 因未使用高相关性上下文而应用的扣分（可通过 unusedHighRelevanceContext 配置）
• 原本可以改善响应的缺失上下文（通过 missingContextPerItem 扣分，最高为 maxMissingContextPenalty）

优化策略

使用结果改进系统：

• 筛除不相关上下文：处理前移除相关性为 low/none 的片段
• 确保使用上下文：确保纳入高相关性上下文
• 填补上下文缺口：添加 Scorer 识别出的缺失信息
• 平衡上下文大小：找到能够获得最佳相关性的适当上下文数量
• 调整扣分敏感度：根据应用对未使用或缺失上下文的容忍度，调整 unusedHighRelevanceContext、missingContextPerItem 和 maxMissingContextPenalty

与 Context Precision 的区别

方面	Context Relevance	Context Precision
算法	带扣分的加权等级	Mean Average Precision (MAP)
相关性	多个等级（high/medium/low/none）	二元值（yes/no）
位置	不考虑	至关重要（奖励靠前的位置）
使用情况	跟踪未使用的上下文并扣分	不考虑
缺失情况	识别缺口并扣分	不评估

Scorer 配置

自定义扣分配置

控制如何对未使用和缺失的上下文应用扣分：

import { createContextRelevanceScorerLLM } from '@mastra/evals'

// Stricter penalty configuration
const strictScorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    context: [
      'Einstein won the Nobel Prize for photoelectric effect',
      'He developed the theory of relativity',
      'Einstein was born in Germany',
    ],
    penalties: {
      unusedHighRelevanceContext: 0.2, // 20% penalty per unused high-relevance context
      missingContextPerItem: 0.25, // 25% penalty per missing context item
      maxMissingContextPenalty: 0.6, // Maximum 60% penalty for missing context
    },
    scale: 1,
  },
})

// Lenient penalty configuration
const lenientScorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    context: [
      'Einstein won the Nobel Prize for photoelectric effect',
      'He developed the theory of relativity',
      'Einstein was born in Germany',
    ],
    penalties: {
      unusedHighRelevanceContext: 0.05, // 5% penalty per unused high-relevance context
      missingContextPerItem: 0.1, // 10% penalty per missing context item
      maxMissingContextPenalty: 0.3, // Maximum 30% penalty for missing context
    },
    scale: 1,
  },
})

const testRun = {
  input: {
    inputMessages: [
      {
        id: '1',
        role: 'user',
        content: 'What did Einstein achieve in physics?',
      },
    ],
  },
  output: [
    {
      id: '2',
      role: 'assistant',
      content: 'Einstein won the Nobel Prize for his work on the photoelectric effect.',
    },
  ],
}

const strictResult = await strictScorer.run(testRun)
const lenientResult = await lenientScorer.run(testRun)

console.log('Strict penalties:', strictResult.score) // Lower score due to unused context
console.log('Lenient penalties:', lenientResult.score) // Higher score, less penalty

动态提取上下文

const scorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    contextExtractor: (input, output) => {
      // Extract context based on the query
      const userQuery = input?.inputMessages?.[0]?.content || ''
      if (userQuery.includes('Einstein')) {
        return [
          'Einstein won the Nobel Prize for the photoelectric effect',
          'He developed the theory of relativity',
        ]
      }
      return ['General physics information']
    },
    penalties: {
      unusedHighRelevanceContext: 0.15,
    },
  },
})

自定义缩放系数

const scorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    context: ['Relevant information...', 'Supporting details...'],
    scale: 100, // Scale scores from 0-100 instead of 0-1
  },
})

// Result will be scaled: score: 85 instead of 0.85

组合多个上下文来源

const scorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    contextExtractor: (input, output) => {
      const query = input?.inputMessages?.[0]?.content || ''

      // Combine from multiple sources
      const kbContext = knowledgeBase.search(query)
      const docContext = documentStore.retrieve(query)
      const cacheContext = contextCache.get(query)

      return [...kbContext, ...docContext, ...cacheContext]
    },
    scale: 1,
  },
})

示例

高相关性示例

此示例展示了极佳的上下文相关性，所有上下文都直接支持响应：

import { createContextRelevanceScorerLLM } from '@mastra/evals'

const scorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    context: [
      'Einstein won the Nobel Prize for his discovery of the photoelectric effect in 1921.',
      'He published his theory of special relativity in 1905.',
      'His general relativity theory, published in 1915, revolutionized our understanding of gravity.',
    ],
    scale: 1,
  },
})

const result = await scorer.run({
  input: {
    inputMessages: [
      {
        id: '1',
        role: 'user',
        content: "What were Einstein's major scientific achievements?",
      },
    ],
  },
  output: [
    {
      id: '2',
      role: 'assistant',
      content:
        "Einstein's major achievements include the Nobel Prize for the photoelectric effect, special relativity in 1905, and general relativity in 1915.",
    },
  ],
})

console.log(result)
// Output:
// {
//   score: 1.0,
//   reason: "The score is 1.0 because all context pieces are highly relevant to Einstein's achievements and were effectively used in generating the comprehensive response."
// }

混合相关性示例

此示例展示了中等相关性，其中部分上下文不相关或未使用：

import { createContextRelevanceScorerLLM } from '@mastra/evals'

const scorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    context: [
      'Solar eclipses occur when the Moon blocks the Sun.',
      'The Moon moves between the Earth and Sun during eclipses.',
      'The Moon is visible at night.',
      'Stars twinkle due to atmospheric interference.',
      'Total eclipses can last up to 7.5 minutes.',
    ],
    scale: 1,
  },
})

const result = await scorer.run({
  input: {
    inputMessages: [
      {
        id: '1',
        role: 'user',
        content: 'What causes solar eclipses?',
      },
    ],
  },
  output: [
    {
      id: '2',
      role: 'assistant',
      content:
        'Solar eclipses happen when the Moon moves between Earth and the Sun, blocking sunlight.',
    },
  ],
})

console.log(result)
// Output with default penalties:
// {
//   score: 0.64,
//   reason: "The score is 0.64 because contexts 1 and 2 are highly relevant and used, context 5 is relevant but unused (10% penalty), while contexts 3 and 4 are irrelevant."
// }

// With custom penalty configuration
const customScorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    context: [
      'Solar eclipses occur when the Moon blocks the Sun.',
      'The Moon moves between the Earth and Sun during eclipses.',
      'The Moon is visible at night.',
      'Stars twinkle due to atmospheric interference.',
      'Total eclipses can last up to 7.5 minutes.',
    ],
    penalties: {
      unusedHighRelevanceContext: 0.05, // Lower penalty for unused context
      missingContextPerItem: 0.1,
      maxMissingContextPenalty: 0.3,
    },
  },
})

const customResult = await customScorer.run({
  input: {
    inputMessages: [{ id: '1', role: 'user', content: 'What causes solar eclipses?' }],
  },
  output: [
    {
      id: '2',
      role: 'assistant',
      content:
        'Solar eclipses happen when the Moon moves between Earth and the Sun, blocking sunlight.',
    },
  ],
})

console.log(customResult)
// Output with lenient penalties:
// {
//   score: 0.69, // Higher score due to reduced penalty for unused context
//   reason: "The score is 0.69 because contexts 1 and 2 are highly relevant and used, context 5 is relevant but unused (5% penalty), while contexts 3 and 4 are irrelevant."
// }

低相关性示例

此示例展示了较差的上下文相关性，其中大部分信息不相关：

import { createContextRelevanceScorerLLM } from '@mastra/evals'

const scorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    context: [
      'The Great Barrier Reef is located in Australia.',
      'Coral reefs need warm water to survive.',
      'Many fish species live in coral reefs.',
      'Australia has six states and two territories.',
      'The capital of Australia is Canberra.',
    ],
    scale: 1,
  },
})

const result = await scorer.run({
  input: {
    inputMessages: [
      {
        id: '1',
        role: 'user',
        content: 'What is the capital of Australia?',
      },
    ],
  },
  output: [
    {
      id: '2',
      role: 'assistant',
      content: 'The capital of Australia is Canberra.',
    },
  ],
})

console.log(result)
// Output:
// {
//   score: 0.26,
//   reason: "The score is 0.26 because only context 5 is relevant to the query about Australia's capital, while the other contexts about reefs are completely irrelevant."
// }

动态提取上下文

在 runtime 根据 run 输入提取上下文：

import { createContextRelevanceScorerLLM } from '@mastra/evals'

const scorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    contextExtractor: (input, output) => {
      // Extract query from input
      const query = input?.inputMessages?.[0]?.content || ''

      // Dynamically retrieve context based on query
      if (query.toLowerCase().includes('einstein')) {
        return [
          'Einstein developed E=mc²',
          'He won the Nobel Prize in 1921',
          'His theories revolutionized physics',
        ]
      }

      if (query.toLowerCase().includes('climate')) {
        return [
          'Global temperatures are rising',
          'CO2 levels affect climate',
          'Renewable energy reduces emissions',
        ]
      }

      return ['General knowledge base entry']
    },
    penalties: {
      unusedHighRelevanceContext: 0.15, // 15% penalty for unused relevant context
      missingContextPerItem: 0.2, // 20% penalty per missing context item
      maxMissingContextPenalty: 0.4, // Cap at 40% total missing context penalty
    },
    scale: 1,
  },
})

RAG 系统集成

与 RAG pipeline 集成，以评估检索到的上下文：

import { createContextRelevanceScorerLLM } from '@mastra/evals'

const scorer = createContextRelevanceScorerLLM({
  model: 'openai/gpt-5.6-sol',
  options: {
    contextExtractor: (input, output) => {
      // Extract from RAG retrieval results
      const ragResults = inputData.metadata?.ragResults || []

      // Return the text content of retrieved documents
      return ragResults.filter(doc => doc.relevanceScore > 0.5).map(doc => doc.content)
    },
    penalties: {
      unusedHighRelevanceContext: 0.12, // Moderate penalty for unused RAG context
      missingContextPerItem: 0.18, // Higher penalty for missing information in RAG
      maxMissingContextPenalty: 0.45, // Slightly higher cap for RAG systems
    },
    scale: 1,
  },
})

// Evaluate RAG system performance
const evaluateRAG = async testCases => {
  const results = []

  for (const testCase of testCases) {
    const score = await scorer.run(testCase)
    results.push({
      query: testCase.inputData.inputMessages[0].content,
      relevanceScore: score.score,
      feedback: score.reason,
      unusedContext: score.reason.includes('unused'),
      missingContext: score.reason.includes('missing'),
    })
  }

  return results
}

与 Context Precision 比较

根据需求选择合适的 Scorer：

用例	Context Relevance	Context Precision
RAG 评估	使用情况很重要时	排序很重要时
上下文质量	细分等级	二元相关性
缺失检测	✓ 识别缺口	✗ 不评估
使用情况跟踪	✓ 跟踪利用情况	✗ 不考虑
位置敏感性	✗ 与位置无关	✓ 奖励靠前的位置

相关内容

• Context Precision Scorer：使用 MAP 评估上下文排序
• Faithfulness Scorer：衡量答案基于上下文的程度
• 自定义 Scorer：创建自己的评估指标