Module 1.1: How AI Training Works

---

Introduction

Every time you interact with an AI assistant like ChatGPT, Claude, or Gemini, you're experiencing the result of thousands of hours of human evaluation work. Behind every helpful response, every refused harmful request, and every well-formatted answer is a training process that relied on people making judgments about what "good" and "bad" AI responses look like.

This module explains how that process works and where you fit into it.

---

Section 1.1.1: RLHF Fundamentals

What is RLHF?

RLHF stands for Reinforcement Learning from Human Feedback. It's the primary method used to make AI models helpful, harmless, and honest after their initial training.

Here's the problem RLHF solves: When AI models are first trained, they learn to predict text by reading billions of web pages, books, and articles. This makes them knowledgeable, but it doesn't make them useful. A model trained only on text prediction might:

• Give correct but unhelpful answers
• Produce harmful content without hesitation
• Ignore what the user actually wants
• Ramble without structure

RLHF fixes this by teaching the model what humans actually prefer.

The Three Stages of RLHF

Stage 1: Supervised Fine-Tuning (SFT)

First, human experts write examples of ideal AI responses. These demonstrations show the model what a good assistant looks like.

Example:

User: What's the capital of France?

Human-written ideal response: The capital of France is Paris. 
It's located in the north-central part of the country along 
the Seine River and has been the capital since the 10th century.

The model learns from thousands of these examples.

Stage 2: Reward Model Training

This is where evaluators like you become essential.

Evaluators are shown the same prompt with multiple AI-generated responses. They rank or compare these responses based on quality. This preference data trains a "reward model", a separate AI that learns to predict which responses humans will prefer.

Example task:

Prompt: "Explain photosynthesis to a 10-year-old"

Response A: "Photosynthesis is the process by which plants 
convert light energy, usually from the sun, into chemical 
energy that can be later released to fuel the plant's 
activities. This process involves chlorophyll..."

Response B: "Plants are like little food factories! They take 
sunlight, water from the ground, and air, and mix them 
together to make their own food. The green parts of plants 
are where this magic happens."

Which response better addresses the prompt? → B

Your judgment (B is better) becomes training data for the reward model.

Stage 3: Reinforcement Learning

Finally, the AI model generates responses and the reward model scores them. The AI learns to produce responses that get higher scores, meaning responses more like what humans preferred.

AI generates response → Reward model scores it → AI adjusts to get higher scores → Repeat millions of times

Why Human Feedback Can't Be Automated

You might wonder: "Why can't AI just evaluate AI?"

Several reasons:

1. Preference is subjective: What's "helpful" depends on context, culture, and individual needs. Humans understand these nuances.

2. Safety requires judgment: Determining if content is harmful often requires understanding intent, context, and real-world consequences.

3. Bootstrapping problem: You can't train an AI to have good judgment using an AI that doesn't have good judgment yet.

4. Novel situations: AI encounters new scenarios constantly. Humans can reason about unfamiliar situations; AI trained only on past data cannot.

This is why the industry needs hundreds of thousands of human evaluators, and why this work is valuable.

Key Concept: The Feedback Loop

Human Preferences → Reward Model → AI Behavior → Better Responses → More Human Feedback → Better Reward Model → ...

Your evaluations don't just affect one response. They shape how the model behaves for millions of future users. A single well-reasoned judgment about why Response A is better than Response B teaches the model something it will apply broadly.

This is why consistency and quality in evaluation matter so much.

---