Advanced Evaluation

LLM-as-a-Judge techniques for evaluating AI outputs. Not a single technique but a family of approaches - choosing the right one and mitigating biases is the core competency.

When to Activate

Building automated evaluation pipelines for LLM outputs
Comparing multiple model responses to select the best one
Establishing consistent quality standards
Debugging inconsistent evaluation results
Designing A/B tests for prompt or model changes
Creating rubrics for human or automated evaluation

Core Concepts

Evaluation Taxonomy

Direct Scoring: Single LLM rates one response on a defined scale.

Best for: Objective criteria (factual accuracy, instruction following, toxicity)
Reliability: Moderate to high for well-defined criteria

Pairwise Comparison: LLM compares two responses and selects better one.

Best for: Subjective preferences (tone, style, persuasiveness)
Reliability: Higher than direct scoring for preferences

Known Biases

Bias	Description	Mitigation
Position	First-position preference	Swap positions, check consistency
Length	Longer = higher scores	Explicit prompting, length-normalized scoring
Self-Enhancement	Models rate own outputs higher	Use different model for evaluation
Verbosity	Unnecessary detail rated higher	Criteria-specific rubrics
Authority	Confident tone rated higher	Require evidence citation

Decision Framework

Is there an objective ground truth?
├── Yes → Direct Scoring (factual accuracy, format compliance)
└── No → Pairwise Comparison (tone, style, creativity)

Quick Reference

Direct Scoring Requirements

Clear criteria definitions
Calibrated scale (1-5 recommended)
Chain-of-thought: justification BEFORE score (improves reliability 15-25%)

Pairwise Comparison Protocol

First pass: A in first position
Second pass: B in first position (swap)
Consistency check: If passes disagree → TIE
Final verdict: Consistent winner with averaged confidence

Rubric Components

Level descriptions with clear boundaries
Observable characteristics per level
Edge case guidance
Strictness calibration (lenient/balanced/strict)

Integration

Works with:

context-fundamentals - Effective context structure
tool-design - Evaluation tool schemas
evaluation (foundational) - Core evaluation concepts

For detailed implementation patterns, prompt templates, examples, and metrics: references/full-guide.md

See also: references/implementation-patterns.md, references/bias-mitigation.md, references/metrics-guide.md

advanced-evaluation

SKILL.md