Before you begin
Set the operating boundary first.
- One narrow task with representative, non-sensitive inputs.
- Two or three model or prompt variants that can run under comparable conditions.
- Task-specific success criteria written before outputs are revealed.
- A named human reviewer who owns the final conclusion.
Step by step
Move from scope to a checked artifact.
Scope one practical task
Write the goal, intended user, constraints, required evidence, unacceptable failure modes, and what a usable result must contain. Keep the task narrow enough that the same review criteria apply to every candidate output.
- Checkpoint
- A reviewer can decide whether an output passes without knowing which model produced it.
- Artifact
- Versioned task packet
Freeze the run conditions
Record the complete prompt, provider, exact model identifier, date, temperature, top-p, token limit, tools, and contextual files. State any provider behavior that prevents exact reproduction.
- Checkpoint
- Another reviewer has enough information to repeat the run or explain why exact reproduction is impossible.
- Artifact
- Run manifest
Capture raw outputs before editing
Store each response as an immutable snapshot and assign a neutral label. Keep raw output separate from annotations, corrections, rewrites, and delivery copy so the evidence cannot silently change after review.
- Checkpoint
- The record exposes exactly what each model returned before a human changed it.
- Artifact
- Labeled output snapshots
Review every output against one rubric
Apply the same task-specific criteria to every candidate. Separate instruction following, factual correctness, evidence quality, uncertainty, and usability. Connect consequential factual claims to current sources and state what each source supports.
- Checkpoint
- Fluency, correctness, evidence, and task fit are scored or described separately.
- Artifact
- Review rubric and claim ledger
Publish a bounded conclusion
Reveal model identity only after the first judgment. State which output best fits this task, what corrections are required, what no output achieved, who approved the review, and which limitations prevent broader conclusions.
- Checkpoint
- The conclusion helps with this task without becoming a permanent model ranking.
- Artifact
- Reviewed decision record
Define the next experiment
Record which missing evidence or failure matters most, then change one important variable: prompt, context, model version, retrieval input, or review criterion. Preserve the original record instead of overwriting it.
- Checkpoint
- The next run can teach something specific rather than merely produce more answers.
- Artifact
- Next-test hypothesis
Deliverables
Keep the work reusable and inspectable.
- Versioned task packet
- Run manifest with exact model IDs and parameters
- Immutable raw output snapshots
- Task-specific rubric and claim ledger
- Reviewed decision record with limitations
- One-variable next experiment
Decision boundaries
What this tutorial does not prove.
- A small controlled comparison supports a decision for one task, not a permanent model leaderboard.
- Provider updates, tools, system instructions, and hidden runtime behavior can change later results.
- Blind review reduces one source of brand bias but does not remove reviewer subjectivity or rubric errors.
- High-impact use requires domain-specific evaluation, qualified oversight, and stronger controls.
Source ledger
Check the current primary guidance.
- NIST AI 600-1: Generative AI Profile
Risk-management reference for context-specific evaluation, governance, measurement, and management.
- RichBay: Conversion uncertainty Case
A published example containing a task packet, captured outputs, evidence ledger, bounded verdict, and review record.