1.0 Modality-Specific Assessment

Study Time: 4 hours Prerequisites: None (this is the first module) Learning Objectives:

• Identify different content modalities in AI evaluation tasks
• Understand why modality-specific criteria are essential for quality evaluation
• Classify tasks by primary and secondary modalities
• Apply modality-aware evaluation frameworks
• Recognize common mistakes in cross-modality assessment

---

Introduction

In AI evaluation work, modality refers to the form or medium through which information is communicated. Understanding modality is important because different modalities require different evaluation approaches.

Why Modality Matters

Consider these two tasks:

Task A: Evaluate if an AI's written explanation of photosynthesis is accurate.

Task B: Evaluate if an AI's spoken audio explanation of photosynthesis is accurate.

Both tasks evaluate the same topic (photosynthesis), but they require fundamentally different assessment criteria:

• Task A criteria focus on: factual accuracy, clarity of writing, logical structure
• Task B criteria also need to consider: pronunciation clarity, speaking pace, tone appropriateness, audio quality, absence of background noise

If you evaluate Task B using only text-focused criteria, you'll miss important quality dimensions. This is a common mistake that causes evaluators to fail quality checks on platforms.

Real-World Impact

Across major AI evaluation platforms, modality-specific evaluation is explicitly required. Evaluators who fail to recognize modality differences:

• Can receive quality check failures from audio/video projects for using generic text criteria
• Fail quality assurance checks for missing modality-specific issues
• Receive lower pay rates due to rework requirements
• Lose access to higher-paying specialized projects

Mastering modality assessment is a career differentiator in AI evaluation.

---

1.0.1 Core Modality Types

Text Modality

Definition: Written language in any form (paragraphs, lists, code, chat messages, etc.)

Evaluation Focus:

• Factual accuracy of content
• Grammatical correctness and spelling
• Logical structure and coherence
• Appropriate tone and formality
• Clarity and conciseness

Example Task: "Evaluate if this AI-written email response appropriately addresses a customer complaint."

Sample Criteria:

• Does the response acknowledge the specific complaint mentioned (delayed shipment)?
• Is the tone professional and empathetic?
• Are all sentences grammatically correct?
• Does the response offer a concrete solution?

---

Audio Modality

Definition: Spoken language or other sound-based content

Evaluation Focus (Content):

• Factual accuracy of spoken information
• Completeness of verbal explanation
• Logical flow of spoken argument

Evaluation Focus (Audio-Specific):

• Clarity: Is speech intelligible and clearly pronounced?
• Pace: Is speaking speed appropriate (not rushed or too slow)?
• Tone: Does vocal tone match the intended emotion/context?
• Audio Quality: Is recording free from background noise, distortion, echoes?
• Prosody: Are emphasis and intonation natural and appropriate?

Example Task: "Evaluate if this AI-generated audio explains how to change a tire."

Sample Criteria:

• Does the audio mention all important steps (jack placement, lug nut loosening, tire removal, new tire installation, lug nut tightening)?
• Is the speaker's pronunciation clear for technical terms ("lug nuts," "jack stand")?
• Is the speaking pace slow enough for listeners to follow instructions?
• Is the audio free from background noise that would distract from instructions?
• Does the speaker's tone convey appropriate caution when discussing safety (jack placement)?

---

Image Modality

Definition: Static visual content (photos, diagrams, charts, screenshots, illustrations)

Evaluation Focus:

• Accuracy: Do visual elements correctly represent information?
• Clarity: Are important details visible and in focus?
• Relevance: Does the image match the stated purpose?
• Composition: Is framing appropriate for the content?
• Technical Quality: Is resolution adequate, lighting sufficient, colors accurate?

Example Task: "Evaluate if this AI-generated diagram correctly illustrates the water cycle."

Sample Criteria:

• Does the diagram show all key processes (evaporation, condensation, precipitation, collection)?
• Are arrows correctly indicating the direction of water movement?
• Are labels clearly legible and positioned near the relevant elements?
• Is the diagram free from visual clutter that would confuse learners?

---

Video Modality

Definition: Moving visual content, often combined with audio

Evaluation Focus (Visual):

• Correct visual representation of actions or concepts
• Appropriate framing and camera stability
• Visibility of important details
• Visual continuity and flow

Evaluation Focus (Audio, if present):

• All audio modality criteria apply
• Synchronization between audio and visual elements

Evaluation Focus (Integration):

• Do audio and visual elements complement each other?
• Does narration accurately describe what's shown?
• Are visual demonstrations timed appropriately with explanations?

Example Task: "Evaluate if this AI-generated tutorial video correctly demonstrates how to fold origami."

Sample Criteria:

• Are hand movements visible and in-frame throughout all folding steps?
• Does the video show each fold clearly before moving to the next step?
• If narration is present, does it accurately describe the fold being demonstrated?
• Is camera angle appropriate to see the folding technique?
• Is video quality sufficient to distinguish between different paper layers?

---

Multimodal Tasks

Definition: Tasks involving multiple modalities simultaneously

Common Combinations:

• Text + Image: AI describes what's in a photo
• Audio + Image: Spoken description of a chart or diagram
• Video + Text: AI generates captions for video content
• Audio + Video: Standard video with narration

Important: Evaluate EACH modality independently AND their integration

Example Task: "Evaluate if this AI-generated audio correctly answers a question about a provided graph."

Required Criteria Types:

1. Image Comprehension: Does the response demonstrate understanding of the graph's data?
2. Factual Accuracy: Are numerical values from the graph stated correctly?
3. Audio Quality: Is the spoken response clear and intelligible?
4. Integration: Does the audio directly reference specific elements visible in the graph?

Common Mistake: Evaluating only content accuracy while ignoring audio quality or visual comprehension.

---