Controlled model comparison — AI Stack

Components and roles

Every component earns its place.

01
Versioned task packet
Freezes the prompt, constraints, and success criteria.
02
Controlled model runs
Captures one immutable output per selected model and parameter set.
03
Evidence ledger
Connects each review claim to a visible source or output snapshot.
04
Human review checkpoint
Owns the final conclusion, limitations, and publication status.

Data flow

Task packet → controlled runs → immutable snapshots → evidence review → published case

Default choices

Start here, then switch for a stated reason.

Task definition

Versioned Markdown or JSON task packet

Selection reason: Keeps the prompt, constraints, criteria, and revision readable and diffable.
Switch when: Use a formal evaluation platform when run volume, access control, or reviewer assignment outgrows repository files.

Evidence record

Immutable output files plus a structured claim ledger

Selection reason: Separates what the model returned from later annotations and conclusions.
Switch when: Use signed object storage when audit retention or multi-team access requires stronger controls.

Operating assumptions

Cost, privacy, alternatives, and limits.

Cost assumption: Inference is usage-based; editorial review time must be budgeted separately. Recheck provider pricing before each batch.
Privacy boundary: The task packet and captured outputs are sent to the selected providers unless a local model is used.
Reference implementation: RichBay task packet + controlled run capture + evidence ledger + reviewed publication contract.
Alternatives: Use one provider or a local model when privacy or reproducibility matters more than breadth.
Known limitation: A small comparison supports a task decision, not a permanent model ranking.

Human checkpoints

Keep accountable decisions visible.

Approve the task and criteria
Inspect every captured output
Own the bounded conclusion

Evidence and sources

See what supports this record.

RichBay tested

Published RichBay Case

Last reviewed 2026-09-05. Recheck current provider documentation, limits, pricing, and terms before adoption.

Every component earns its place.

Versioned task packet

Freezes the prompt, constraints, and success criteria.

Controlled model runs

Captures one immutable output per selected model and parameter set.

Evidence ledger

Connects each review claim to a visible source or output snapshot.

Human review checkpoint

Owns the final conclusion, limitations, and publication status.

Data flow

Task packet → controlled runs → immutable snapshots → evidence review → published case

Start here, then switch for a stated reason.

Task definition

Versioned Markdown or JSON task packet

Selection reason: Keeps the prompt, constraints, criteria, and revision readable and diffable.
Switch when: Use a formal evaluation platform when run volume, access control, or reviewer assignment outgrows repository files.

Evidence record

Immutable output files plus a structured claim ledger

Selection reason: Separates what the model returned from later annotations and conclusions.
Switch when: Use signed object storage when audit retention or multi-team access requires stronger controls.

Cost, privacy, alternatives, and limits.

Cost assumption

Inference is usage-based; editorial review time must be budgeted separately. Recheck provider pricing before each batch.

Privacy boundary

The task packet and captured outputs are sent to the selected providers unless a local model is used.

Reference implementation

RichBay task packet + controlled run capture + evidence ledger + reviewed publication contract.

Alternatives

Use one provider or a local model when privacy or reproducibility matters more than breadth.

Known limitation

A small comparison supports a task decision, not a permanent model ranking.