1.5 Prompt Engineering for Evaluation Tasks

Study Time: 4.5 hours Prerequisites: Module 1.1 Learning Objectives:

• Understand why prompt quality matters for AI training
• Recognize prompt layers and the instruction hierarchy
• Apply the prompt completeness checklist to assess prompts
• Identify explicit, implicit, and negative constraints in prompts
• Master the iterative failure-finding loop for prompt creation
• Write natural, specific prompts that meet quality standards
• Handle multimodal prompt grounding requirements
• Apply multi-turn strategic planning techniques
• Avoid common prompt writing mistakes

---

Introduction

"The purpose of prompts is to identify vulnerable aspects of knowledge and reasoning in LLMs"

Prompts are the test cases that expose model strengths and weaknesses. Poor prompts lead to poor training data, which leads to poorly trained models. Your prompts directly shape the AI systems millions will use.

This module covers everything you need to know about writing and evaluating prompts for AI training tasks.

---

1.5.1 Why Prompt Quality Matters

The Golden Rule

A good prompt reflects what a REAL HUMAN would actually need, not a contrived test designed to trick the model.

Failure-Oriented Design Principle

The entire purpose of prompts is to expose where AI models fail. Training value comes from meaningful failures, not from prompts the model handles easily.

Key Insight: If the model handles your prompt well, the prompt wasn't hard enough. Go back and revise.

---

1.5.2 Prompt Layers & the Instruction Hierarchy

Understanding Where Your Prompt Fits

When a user sends a prompt to an AI model, that prompt is not processed in isolation. It exists within a hierarchy of instructions — layers of context that shape how the model responds.

Understanding this hierarchy is foundational. It affects how you write prompts, how you evaluate responses, and how you handle situations where different instructions conflict.

---

The Three Operational Layers

In practice, AI models operate with three primary layers of instructions:

Layer	Name	Who Writes It	What It Controls
Layer 1	Core Model Instructions (CMI)	The AI provider	Safety boundaries, fundamental behavior rules, ethical guidelines
Layer 2	Model Customization Instructions (MCI)	The developer/deployer	Role, tone, domain constraints, specific capabilities
Layer 3	User Prompt	The end user	The specific task, question, or request

Layer 1 — Core Model Instructions: These are the foundational rules baked into the model by its creator. They define what the model will and will not do regardless of other instructions. For example, safety guidelines that prevent the model from providing instructions for harmful activities. Users and developers typically cannot see or modify this layer.

Layer 2 — Model Customization Instructions: These are instructions set by the developer who deploys the model for a specific use case. They define the model's role, tone, and boundaries for that particular application. For example, "You are a customer service assistant for a software company. Only answer questions related to our products." Different providers use different terminology — "system prompts," "developer messages," or "system instructions."

Layer 3 — User Prompt: This is the input from the person actually using the model — the question, task, or conversation turn. This is the layer evaluators work with most directly.

---

Additional Layers

Beyond these three operational layers, two additional layers exist:

• Alignment Layer (below Layer 1): The foundational behavioral traits the model learned during its initial training process. These are intrinsic to the model and invisible in any prompt.
• Tool & Data Outputs (beyond Layer 3): Content from external tools like web search results, database queries, or file retrieval. This content is treated as untrusted data — it informs the response but should not override higher-layer instructions.

For evaluation purposes, the three operational layers are what you will encounter most often.

---

Priority and Conflicts

The layers follow a priority order: Layer 1 > Layer 2 > Layer 3.

When instructions at different layers conflict, the model should follow the higher-priority layer. For example:

• If Layer 2 says "Always be concise" and the user asks for a detailed explanation, the model must navigate this tension
• If the user asks the model to do something that violates Layer 1 safety guidelines, the model should follow Layer 1

These conflicts between layers are called tensions. You will learn to evaluate how models handle tensions in Level 2 (Module 9.1). For now, understanding that these layers exist and have a priority order is the key foundation.

---

Why This Matters for Evaluation

As an evaluator, you need to consider:

1. Which layer does each instruction come from? — A requirement in the system prompt (Layer 2) takes priority over a user's attempt to override it
2. Is the model correctly prioritizing? — When layers conflict, the model should follow the higher layer
3. Are you evaluating against the right layer? — Your rubric criteria may reference requirements from different layers

---