AI Evaluator Certification: the full curriculum

Core Competencies of AI Evaluators

Master the 4 core competencies of AI evaluators and learn cognitive load management strategies for consistent evaluation work.

• Understand the 4 core competencies that define successful AI evaluators
• Learn to manage cognitive load for sustained evaluation quality
• Develop systematic approaches to evaluation work

Topics: four core competencies, five skills of AI evaluation work, instruction following, error detection, clear feedback

Modality-Specific Assessment

Evaluate AI outputs across text, code, image, and multi-modal modalities. Recognize modality-specific quality cues and failure patterns.

• Identify different content modalities in AI evaluation tasks
• Understand why modality-specific criteria are essential
• Classify tasks by primary and secondary modalities

Topics: modality, text evaluation, audio evaluation, image evaluation, video evaluation

How AI Training Works

Learn about RLHF, the three stages of AI training, different evaluation task types, and the quality standards evaluators are held to.

• Understand what RLHF is and why it matters
• Identify the three stages of RLHF
• Recognize different types of evaluation tasks

Topics: RLHF, reinforcement learning from human feedback, supervised fine-tuning, reward model training, evaluation task types

Core Evaluation Skills

Master comparison methodology and reading & interpreting rubrics effectively, applying guidelines consistently.

• Apply the five-step systematic comparison framework instead of trusting a first-read impression
• Set the two or three decision criteria that matter for a prompt before reading any response
• Rank three or more responses with a consistent K-way strategy and resolve close calls

Topics: comparison methodology, rubric reading, rubric interpretation, systematic comparison, decision making

Response Quality Assessment

Learn to evaluate AI responses across key dimensions: accuracy, hallucination detection, instruction following, and failure type hierarchy.

• Score helpfulness against the user's intent rather than the literal question, and spot pleasantry and repetition anti-patterns
• Classify accuracy errors (factual, logical, outdated, hallucination, precision) and trace each to a recognition, reasoning or calculation origin
• Evaluate accuracy on a topic outside your own expertise

Topics: five core dimensions, helpfulness, accuracy, safety, instruction following

Safety Fundamentals

Understand AI safety principles, harm categories, and how to evaluate responses for safety compliance.

• Recognize the six harm categories: physical, psychological, financial, reputational, societal and legal
• Weigh severity against likelihood and give vulnerable populations extra weight
• Spot the five red flags for unsafe content and the common disguises: educational, hypothetical, role-play and incremental framing

Topics: AI safety, harm categories, harmful content, safety evaluation, risk assessment

Prompt Engineering for Evaluation Tasks

Master prompt decomposition, quality criteria identification, and structured evaluation approaches for different prompt types.

• Understand why prompt quality matters for AI training
• Recognize prompt layers and the instruction hierarchy
• Apply the prompt completeness checklist to assess prompts

Topics: prompt quality, instruction hierarchy, prompt layers, core model instructions, model customization instructions

Justification Writing

Write clear, defensible justifications with the four-part structure: conclusion, criteria, evidence, connection. Master evidence-based reasoning for evaluation decisions.

• Write evidence-based evaluation rationales
• Structure effective justifications using the dimension reference framework
• Apply the Pyramid Principle to lead justifications with the verdict

Topics: justification structure, evidence-based reasoning, pyramid principle, decisive dimensions, deviation

Data Annotation Fundamentals

Master annotation taxonomies, labeling guidelines, and quality control for structured AI training data.

• Understand the connection between data annotation and AI evaluation
• Master image annotation types
• Apply text annotation techniques

Topics: image/text/audio/video annotation, IoU, inter-annotator agreement, flag vs skip, annotation quality metrics

Ideal Response Description & Rubric Properties

Create Ideal Response Descriptions through prompt analysis. Master criterion properties: atomicity, self-containment, objectivity, specificity, weighting, and golden responses.

• Understand what rubrics are and how their structure varies across projects
• Identify the components a rubric may include and their purpose
• Create Ideal Response Descriptions (IRDs) through systematic prompt analysis

Topics: rubric definition, analytic rubrics, holistic rubrics, rubric components, rationale

Atomicity Bootcamp

Recognize and decompose non-atomic criteria. Write atomic criteria from scratch and apply atomicity reliably across any task type.

• Define atomicity in the context of evaluation criteria
• Recognize non-atomic criteria instantly (within 30 seconds)
• Decompose multi-aspect criteria into atomic components

Topics: atomicity, atomic criteria, non-atomic criteria, criterion decomposition, the AND test

Instance-Specific Mastery

Write criteria tied to the specific instance, not to generic task templates. Surface what this prompt requires that others would not.

• Distinguish between generic and instance-specific criteria
• Write criteria that reference actual task content
• Avoid a leading cause of evaluation quality failures

Topics: instance-specificity, instance-specific criteria, prompt-specific requirements, generic vs specific criteria, template criteria

Self-Containment

Write criteria a reviewer can apply without consulting external materials. Eliminate hidden dependencies and implicit context.

• Define self-containment in evaluation criteria
• Recognize cross-referencing violations
• Write criteria that stand alone

Topics: self-containment, standalone criteria

Objectivity & Thresholds

Convert subjective judgments into concrete thresholds. Build criteria two reviewers would score the same way.

• Understand the objectivity-subjectivity spectrum and why rubric projects lean objective
• Distinguish between objective and subjective criteria
• Write measurable criteria with concrete thresholds

Topics: objectivity, thresholds, subjective vs objective criteria, measurable thresholds, scoring consistency

Modality-Aware Rubrics

Apply rubric properties to real tasks. Master modality-aware evaluation, platform patterns, speed drills, and integration practice.

• Integrate all five core properties into complete rubrics
• Build modality-specific criteria for audio and video tasks
• Apply the Complete Rubric Checklist before every submission

Topics: modality-aware criteria, the five core properties, audio criteria, video criteria, multimodal integration

Platform Rubric Patterns

Recognize the rubric patterns each major platform expects. Adapt your rubric writing to platform conventions without losing rigor.

• Recognize common rubric patterns across platforms
• Adapt evaluation approach to different platform styles
• Build platform-agnostic transferable skills

Topics: platform rubric patterns, rubric adaptation

Rubric Speed Drills

Build the muscle to draft sound rubrics under platform-realistic time limits. Cut deliberation time without dropping rubric quality.

• Build rubrics quickly while maintaining quality
• Develop muscle memory for criteria writing
• Know your pace per criterion so any rubric can be budgeted by its size, and follow platform time guidance

Topics: rubric speed, speed drills, rubric efficiency

Integration Practice

Combine rubric writing, evaluation, and justification on end-to-end tasks that mirror real platform workflows.

• Apply all five properties simultaneously
• Build complete certification-ready rubrics
• Handle complex real-world scenarios

Topics: five core properties, complete rubric checklist, multimodal rubrics, handling ambiguity, integration practice

Citation Speed Training

Find and format citations rapidly. Master source reliability evaluation and platform citation formats.

• Find citations rapidly (5 claims in 10 minutes)
• Verify factual accuracy efficiently
• Use reliable sources quickly

Topics: citation, fact-checking, source reliability, verification workflow, source evaluation

Source Reliability

Distinguish strong, weak, and unreliable sources. Apply a repeatable check to every citation before you accept it as evidence.

• Evaluate source credibility quickly
• Identify bias and conflicts of interest
• Distinguish primary from secondary sources

Topics: primary vs secondary sources, bias and conflicts of interest, red flags, Triangle Method

Platform Citation Formats

Format citations to match each platform's expected style. Move fast without tripping on platform-specific format rules.

• Format citations correctly for different platforms
• Meet platform-specific citation requirements
• Build consistent citation habits

Topics: citation formats, platform citation styles, citation formatting

Platform Navigation & Tools

Navigate platform interfaces and tooling efficiently. Reduce friction so your evaluation time goes into the work, not the UI.

• Navigate platform interfaces efficiently
• Master common platform tools and features
• Understand task queues and workflows

Topics: platform navigation, platform interfaces, common platform tools, task queues, workflow optimization

Time Management & Productivity

Run an evaluation session with a sustainable cadence. Protect attention across the hours where careless mistakes pile up.

• Budget time effectively for different task types
• Make the most of your working time through efficiency
• Maintain consistent quality under time pressure

Topics: time management, productivity, fatigue

Gating Test Simulations

Walk into the gating exam knowing what to expect. Practice on representative tasks under representative time pressure.

• Understand gating test formats and requirements
• Practice certification-level evaluation scenarios
• Master test-taking strategies for qualification exams

Topics: gating tests, gating exam, exam preparation, time pressure, qualification tests