Experiment · Controlled output comparison

Does this conversion result prove an improvement?

Hold the task and parameters fixed, vary the model, and compare how each handles causal uncertainty.

Task Hold the prompt and settings fixedAction Change the compared model runsResult Inspect changes and conclusion limits

Controlled variable: Model output; prompt and generation settings stay fixed

Experiment setup

Hold the task fixed. Change the model run.

Version A converted 8 of 100 visitors. Version B converted 14 of 100 visitors during the following seven days. No randomized split was used, and traffic sources differed between the weeks. Does this prove B is better? Give a decision recommendation and identify the uncertainty.

Changed variable
Resolved model ID
Fixed conditions
Prompt · max 2000 tokens · temperature 0.2 · top-p 0.9
Evidence set
Three captured outputs from one versioned generation batch

Change the variable

Select two captured runs to compare.

Try more than one pairing. The output and task-specific review update together.

Run A

OpenAI

meets

openai/gpt-oss-20b

No – the data do not prove that Version B is superior.

Why not?

  • The two weeks were not randomized; traffic sources (e.g., referral, paid, organic) differed, so any conversion difference could be due to those sources rather than the version.
  • The sample size is small (100 visitors per week), so the observed 8 % vs. 14 % difference could be due to random variation.
  • External factors (seasonality, day‑of‑week, marketing campaigns) could have changed between the weeks.

Decision recommendation
Run a properly randomized A/B test (or multivariate test) with a larger, balanced sample and control for traffic source. Only then can you statistically assess whether B truly outperforms A.

Uncertainty

  • Confounding by traffic source and other time‑varying factors.
  • Statistical uncertainty due to the small sample size.
  • Potential interaction with other variables (device, location, etc.) that were not controlled.
Inspect this run’s review
Strengths
  • Names both statistical uncertainty and confounding
  • Recommends a balanced randomized test
  • Labels additional possible confounders as hypotheses rather than observed events
Risks
  • The possible confounders remain unverified; collect evidence before attributing the difference to any of them

Best use: A decision memo that must explain uncertainty.

Run B

Qwen

meets

qwen/qwen3-30b-a3b-fp8

No, this does not prove B is better. The lack of a randomized split and differing traffic sources introduce confounding variables. Recommendation: Conduct a controlled experiment with random assignment. Uncertainty: Results may be influenced by external factors.

Inspect this run’s review
Strengths
  • Directly rejects proof
  • Identifies confounding and random assignment
Risks
  • Does not mention the small sample explicitly

Best use: A concise executive answer.

Bounded conclusion

State only what this run set supports.

All three reject a causal conclusion. They differ in how clearly they identify sample uncertainty and week-to-week confounding.

Evaluation target

Strong answers say the observation does not prove causality, identify small samples and time/traffic confounding, and recommend a randomized concurrent test before a durable rollout.

Reusable comparison method
  1. Separate observed rates from causality.
  2. Name non-random allocation and changing traffic sources.
  3. Recommend a concurrent randomized comparison.

Three captured outputs under one prompt and parameter set; reviewed Sep 9, 2026. This experiment describes these runs only and is not a global model ranking.

Build the next step

Turn this result into a reusable method.

A conclusion boundary for sequential conversion comparisons.

Inspect the Reviewed CaseLearn the review method