2.4 Objectivity & Thresholds

Study Time: 2 hours Prerequisites: Module 2.3 (Self-Containment) Learning Objectives:

• Understand the objectivity-subjectivity spectrum and why rubric projects lean objective
• Distinguish between objective and subjective criteria
• Write measurable criteria with concrete thresholds
• Eliminate vague qualifiers from evaluation
• Use quantitative standards where possible
• Handle subjective elements using illustrative language and outcome-focused criteria
• Build consistency into evaluation criteria

---

Introduction

The fourth critical property of evaluation criteria is objectivity with clear thresholds.

Objective Criterion: Uses concrete, measurable standards that minimize interpretation differences.

Subjective Criterion: Uses vague qualifiers that different evaluators interpret differently.

Why It Matters

Subjective criteria cause:

• Inconsistent scoring: Evaluators have different standards for "good," "sufficient," "adequate"
• Unreliable data: Platform can't use inconsistent evaluations
• Evaluator disagreement: Quality checks fail when evaluators score differently
• Unusable feedback: "Somewhat accurate" doesn't tell creators what to fix

Platforms require objective, measurable criteria for reliable evaluation.

---

2.4.1 The Objectivity-Subjectivity Spectrum

Why Rubric Projects Lean Objective

Rubric-based evaluation projects generally lean toward objectivity. This is not arbitrary — rubrics function best when criteria are objective, verifiable, and evaluable. Objective criteria enable consistent scoring across evaluators, which is the entire purpose of using a rubric.

However, projects exist on a spectrum from fully objective to involving subjective elements. Understanding where a project falls on this spectrum helps you write appropriate criteria.

The Spectrum

Position	Characteristics	Criteria Style
Fully Objective	Closed prompts, no ambiguity, binary outcomes	"Does the response include exactly 3 examples?"
Mostly Objective	Clear requirements with some quality judgment	"Does the response address the user's question directly?"
Mixed	Defined requirements with quality-dependent elements	"Does the explanation build from simple to complex?"
Subjective Elements	Open-ended tasks, quality assessment, creative evaluation	"Does the response demonstrate appropriate creativity for the task?"

Handling Subjective Elements

When a project includes criteria that involve some degree of judgment:

Use illustrative language — Show what good looks like using "such as" or "e.g." rather than prescribing the exact method:

• Over-prescriptive: "The response must use the Socratic method with exactly 3 questions"
• Illustrative: "The response should engage the learner (e.g., through questions, examples, or analogies)"

Evaluate outcomes, not methods — Define what the result should achieve, not how the model should get there:

• Method-focused: "The response must start with a definition, then give 2 examples, then summarize"
• Outcome-focused: "The response should clearly explain the concept so that a beginner could understand it"

Make subjective criteria self-contained and evaluable — Even when perfect objectivity isn't possible, the criterion should give evaluators enough information to score consistently:

• Vague: "Is the response good?"
• Self-contained: "Does the response provide a clear, actionable answer to the user's question (addresses the specific scenario, includes concrete next steps)?"

The Key Balance

The goal is not to eliminate all subjectivity — some tasks genuinely require judgment. The goal is to minimize unnecessary subjectivity and ensure that any remaining subjective elements are well-defined enough for consistent evaluation.

Rule of thumb: If two competent evaluators would likely disagree on the score more than 20% of the time, the criterion needs more objectivity.

---

2.4.2 Objective vs Subjective

The Core Difference

Subjective: "Is the response fairly detailed?"

• What's "fairly"? 2 sentences? 5 sentences? 2 paragraphs?
• Different evaluators = different standards

Objective: "Does the response include at least 3 specific examples?"

• Clear threshold: 3 examples
• Any evaluator can count and verify

---

Common Subjective Qualifiers

These words introduce subjectivity:

• Somewhat, fairly, quite, rather, relatively
• Adequate, sufficient, appropriate, reasonable
• Good, well, properly, effectively
• Too much, too little, enough, many

Without concrete definitions, these are subjective.

---