What is AI evaluation?
Three different professions use the same two words. This page sets them side by side, so you know which one you are reading about, which one you want, and where each is learned.
AI evaluation, AI training, data annotation
When a platform posts work, it may call it any of the three. The person, the platform and the task are the same; the word only says which part of the work is in view.
AI evaluation
Judging model responses: grading against a rubric, comparing outputs, ranking, flagging what is unsafe. The word this certification uses.
Among engineers it also means automated testing of a model, the meaning separated below.
AI training
The platform word: your judgments are the human feedback that trains the model (RLHF). AI trainer is the job title platforms such as Handshake AI use for the work.
Among engineers it also means training a model with code and data, a different job.
Data annotation
The older word: labelling text, images or audio so a model can learn from them, which today mostly means rating and comparing model responses. Still the name of a platform and of many job posts.
It also names the dataset-labelling industry that supplies AI labs.
The same words, three different jobs
Search results and AI assistants mix these together. Each column is a different profession with its own titles, tools and training.
Human AI evaluation
- Who does it
- People who grade, compare and annotate AI responses so labs can train and check their models.
- Typical titles
- AI evaluator, AI trainer, AI rater, LLM trainer, AI tutor, data annotator.
- What the work is
- Read the guidelines, judge a response against a rubric, compare two outputs, write a justification, flag unsafe content.
- Tools
- A platform's task interface and your own written reasoning.
- Coding needed
- No coding. Reading, writing and judgment, with subject knowledge for specialist queues.
- Where to learn it
- The AI Evaluator Certification below; the first module is free.
Technical LLM evaluation
- Who does it
- Engineers measuring how well a model or an AI application performs.
- Typical titles
- ML engineer, AI engineer, evaluation engineer, applied scientist.
- What the work is
- Build test sets, define metrics, run benchmarks, automate judging with another model (LLM-as-a-judge), track regressions.
- Tools
- Evaluation frameworks, Python, benchmark suites, observability dashboards.
- Coding needed
- Yes. This is software and data work.
- Where to learn it
- Developer courses and the documentation of the tools themselves. Not taught here.
Program evaluation
- Who does it
- Evaluators in the monitoring-and-evaluation profession assessing whether a program met its goals, sometimes a program that used AI.
- Typical titles
- M&E specialist, program evaluator, impact analyst.
- What the work is
- Design indicators, collect evidence, judge outcomes against objectives, report to funders.
- Tools
- Logic models, surveys, statistics, reporting frameworks.
- Coding needed
- Usually not; research methods and statistics.
- Where to learn it
- Public-policy and evaluation programs and professional evaluation associations. Not taught here.
Which words belong to which field
If a page talks about frameworks, metrics and benchmarks, it is written for engineers. If it talks about rubrics, queues, guidelines and justifications, it is about the job.
The human side, one job with many titles
Platforms post the same work under different names. These all mean a person judging AI output.
- AI evaluator
- AI trainer
- AI rater
- LLM trainer
- AI tutor
- Data annotator
- Human feedback (RLHF) contributor
- Response grader
The engineering side
Words you will meet in developer material. They describe automated or systematic testing of models, not a job you can do without code.
- Evals (as engineers use it)
- LLM-as-a-judge
- Benchmark
- Evaluation framework
- Metrics
- Test set
- Regression
Words both sides use
Rubric, grading, ground truth and quality mean the same thing in both fields; who applies them differs.
- Rubric
- Grading
- Ground truth
- Inter-annotator agreement
- Hallucination
- Safety
What human AI evaluation involves
Five skills, tested by every platform's guidelines and assessments in one form or another. The certification teaches each one; a module number is given for each.
Prompt engineering for evaluation tasks
Writing complex, well-specified prompts: instruction hierarchy, constraint types, completeness, multi-turn planning.
Drafting the ideal response
Describing what a correct, complete answer looks like before judging any candidate.
Designing accurate, objective rubrics
Atomic criteria, self-containment, objectivity and thresholds, modality-aware and platform-specific rubric patterns.
Grading
Applying a rubric consistently across cases that do not fit neatly; rating systems, comparison methodology, response-quality dimensions, safety.
Writing objective justifications
Defending each call in writing in the terms the guidelines use, with citations and source reliability.
The full module list, with what each module teaches, is on the curriculum page. For the career itself, read how to become an AI evaluator, how to become an AI trainer with no experience and what an AI evaluator certification is. Glossary: AI evaluator, AI evals and human evaluation in AI.
Is there a data annotation certification?
The honest answer first, then what the work actually tests and where this program fits.
No platform requires a data annotation certification. DataAnnotation, Outlier, Mercor, Alignerr and the other platforms hire through their own applications and unpaid qualification tests, and those tests check instruction-following, rubric application and written reasoning, not a credential. A certification is useful only if it teaches those three things.
Most data annotation work advertised today is the rubric-and-text side of the job: reading guidelines, grading and comparing model responses, annotating what is wrong and explaining the call in writing. That is the same work platforms post as AI evaluation or AI training, and it is what the AI Evaluator Certification trains: 24 modules on the five skills above, with Kappa, the AI tutor, critiquing your reasoning in every lesson, then a proctored, ID-verified exam and a certificate verifiable by code. It does not teach image, video or point-cloud labelling tools, which some annotation vendors use for computer-vision work.
If you want to see what the platforms' own tests look like first, read how to pass the DataAnnotation assessment and how to become an AI trainer with no experience, which lists what each platform's qualification test contains.
Pick the meaning that matches your goal
You want the job: grading AI, remote, no coding
That is human AI evaluation. Learn the five skills, then apply to platforms with a credential that shows you can do the work. Start with the free module.
Start the free moduleYou build AI systems and need to test them
That is technical LLM evaluation. Learn it through developer courses and the documentation of the evaluation tools you will use. This site does not teach it.
You assess whether a program worked
That is program evaluation, a profession of its own with its own training and associations. This site does not teach it either.
What Annotation Academy is, and is not
Three different fields use the words AI evaluation. This program is for one of them: the people who grade AI, not the people who build it.
It is
- Training and certification for human AI evaluation, AI training and data annotation work: the AI evaluator, AI trainer, AI rater and data annotator roles on platforms such as Outlier, DataAnnotation, Alignerr, Ethos, Mercor and Micro1.
- Five skills: prompt engineering for evaluation tasks, drafting the ideal response, designing objective rubrics, grading, and writing objective justifications.
- Independent: not affiliated with any platform or AI lab. The certificate is earned through an ID-verified, proctored exam and is verifiable by its code.
- Open to beginners: no coding or AI background needed. The first module is free.
It is not
- An ML, data-science or coding course. We do not teach model building, fine-tuning or evaluation frameworks.
- An LLM app-development course or a prompt-engineering course for developers.
- A general AI-literacy course.
- A job-placement service. Platforms do not require the certification; it trains the skills the work requires, which is also what platform assessments test.
Common questions
Is AI training the same as AI evaluation?
On a platform, yes. AI training jobs are the human-feedback work of judging and correcting model responses, and AI trainer is the job title many platforms use for it. Among engineers, AI training means training a model with data and code, which is a different job. This certification trains the human work.
Is data annotation the same as AI evaluation?
It is the older name for the same family of work: labelling text, images or audio so a model can learn from them, which today mostly means rating and comparing model responses. Platforms post the work under both names. The skills overlap almost entirely; the certification covers the evaluation side in depth and the annotation fundamentals.
Is there a data annotation certification?
No platform requires one. DataAnnotation, Outlier, Mercor, Alignerr and the others hire through their own applications and qualification tests, which check instruction-following, rubric application and written reasoning rather than a credential. The AI Evaluator Certification is an independent, ID-verified, proctored credential for that same work: the rubric-and-text side of data annotation (grading, comparing and annotating model responses), not image or point-cloud labelling tools. The first module is free.
Is AI evaluation a technical job?
It depends which meaning you want. Human AI evaluation, the AI evaluator or AI trainer job on platforms, is judgment work: reading guidelines, grading responses, comparing outputs and writing justifications, with no coding. Technical LLM evaluation is engineering work with code, metrics and benchmarks.
Is an AI evaluator the same as an AI trainer?
Yes, in the platform sense. AI evaluator, AI trainer, AI rater and LLM trainer are different names for the same human-feedback work. The title a platform uses says nothing about the task; the guidelines do.
Do I need to know Python to become an AI evaluator?
No. Human evaluation work needs careful reading, clear writing and consistent judgment, plus subject knowledge for specialist queues such as coding, law or medicine. Python is for the engineering side of evaluation.
What does an AI evaluator actually do all day?
Works through tasks in a platform interface: reads a prompt and one or more model responses, checks them against a rubric, scores or ranks them, writes a short justification, and flags anything unsafe. Tasks come in queues that change with the platform's projects.
Is there training for AI evaluators?
Yes. The AI Evaluator Certification teaches the five skills the work requires, which is also what platform assessments test: prompt engineering for evaluation tasks, drafting the ideal response, designing objective rubrics, grading, and writing objective justifications, with an AI tutor critiquing your reasoning in every lesson. It is independent, ID-verified and proctored; no platform requires it, and the first module is free.
If you want the job, this is the field.
The first module is free, with no card required. See how the work is taught, then decide.