Core Competencies of AI Evaluators

Master the 4 core competencies of AI evaluators and learn cognitive load management strategies for consistent evaluation work.

Modality-Specific Assessment

Evaluate AI outputs across text, code, image, and multi-modal modalities. Recognize modality-specific quality cues and failure patterns.

How AI Training Works

Learn about RLHF, the three stages of AI training, different evaluation task types, and the HHH framework.

Core Evaluation Skills

Master comparison methodology and reading & interpreting rubrics effectively, applying guidelines consistently.

Response Quality Assessment

Learn to evaluate AI responses across key dimensions: accuracy, hallucination detection, instruction following, and failure type hierarchy.

Safety Fundamentals

Understand AI safety principles, harm categories, and how to evaluate responses for safety compliance.

Prompt Engineering for Evaluation Tasks

Master prompt decomposition, quality criteria identification, and structured evaluation approaches for different prompt types.

Justification Writing

Write clear, defensible justifications using the SPEC framework. Master evidence-based reasoning for evaluation decisions.

Data Annotation Fundamentals

Master annotation taxonomies, labeling guidelines, and quality control for structured AI training data.

Ideal Response Description & Rubric Properties

Create Ideal Response Descriptions through prompt analysis. Master criterion properties: atomicity, self-containment, objectivity, specificity, weighting, and golden responses.

Atomicity Bootcamp

Recognize and decompose non-atomic criteria. Write atomic criteria from scratch and apply atomicity reliably across any task type.

Instance-Specific Mastery

Write criteria tied to the specific instance, not to generic task templates. Surface what this prompt requires that others would not.

Self-Containment

Write criteria a reviewer can apply without consulting external materials. Eliminate hidden dependencies and implicit context.

Objectivity & Thresholds

Convert subjective judgments into concrete thresholds. Build criteria two reviewers would score the same way.

Modality-Aware Rubrics

Apply rubric properties to real tasks. Master modality-aware evaluation, platform patterns, speed drills, and integration practice.

Platform Rubric Patterns

Recognize the rubric patterns each major platform expects. Adapt your rubric writing to platform conventions without losing rigor.

Rubric Speed Drills

Build the muscle to draft sound rubrics under platform-realistic time limits. Cut deliberation time without dropping rubric quality.

Integration Practice

Combine rubric writing, evaluation, and justification on end-to-end tasks that mirror real platform workflows.

Citation Speed Training

Find and format citations rapidly. Master source reliability evaluation and platform citation formats.

Source Reliability

Distinguish strong, weak, and unreliable sources. Apply a repeatable check to every citation before you accept it as evidence.

Platform Citation Formats

Format citations to match each platform's expected style. Move fast without tripping on platform-specific format rules.

Platform Navigation & Tools

Navigate platform interfaces and tooling efficiently. Reduce friction so your evaluation time goes into the work, not the UI.

Time Management & Productivity

Run an evaluation session with a sustainable cadence. Protect attention across the hours where careless mistakes pile up.

Gating Test Simulations

Walk into the gating exam knowing what to expect. Practice on representative tasks under representative time pressure.