Experiment · Source-quality stress test

Which source should control the decision?

Hold the source packet fixed, vary the model, and inspect how each resolves conflicting evidence.

Task Hold the prompt and settings fixedAction Change the compared model runsResult Inspect changes and conclusion limits

Controlled variable: Model output; source packet and generation settings stay fixed

Experiment setup

Hold the task fixed. Change the model run.

An evidence packet contains three claims about the fictional Harbor API: (1) its official reference, updated August 20, 2026, says the default limit is 100 requests per minute; (2) a vendor blog from 2025 says 60; (3) an unverified forum post says 200. What limit should an integration use today, and what should the team verify before launch? Use only this packet.

Changed variable
Resolved model ID
Fixed conditions
Prompt · max 2000 tokens · temperature 0.2 · top-p 0.9
Evidence set
Three captured outputs from one versioned generation batch

Change the variable

Select two captured runs to compare.

Try more than one pairing. The output and task-specific review update together.

Run A

OpenAI

meets

openai/gpt-oss-20b

Use 100 requests per minute – the most authoritative source in the packet is the official Harbor API reference (updated 20 Aug 2026).

What to verify before launch

  1. Confirm the reference is still current – check that no newer version of the API documentation has changed the limit.
  2. Test the rate‑limit enforcement – send bursts of requests from your integration to see how the API responds (e.g., 429 status, retry‑after header).
  3. Check for endpoint‑specific or account‑type limits – some endpoints or user tiers may have different caps or burst allowances.
  4. Determine the scope of the limit – is it per API key, per IP, or global?
  5. Validate burst capacity – many APIs allow a short burst above the nominal rate; confirm the maximum burst size.
  6. Ensure your integration’s traffic pattern stays within the limit – plan for peak usage and add back‑off logic if needed.

By following these checks, you’ll be confident that the 100 req/min limit is correct and that your integration will operate reliably.

Inspect this run’s review
Strengths
  • Explains source priority
  • Adds scope, burst, and retry checks
Risks
  • Suggests burst testing that must respect provider rules

Best use: An implementation readiness checklist.

Run B

Qwen

meets

qwen/qwen3-30b-a3b-fp8

The integration should use 100 requests per minute as the default limit, based on the official reference. Before launch, the team should verify if there have been any updates to the official documentation since August 20, 2026, and confirm the current limit with the vendor.

Inspect this run’s review
Strengths
  • Chooses the supported limit
  • Asks for a freshness and vendor check
Risks
  • Does not mention endpoint or account scope

Best use: A concise operational decision.

Bounded conclusion

State only what this run set supports.

All answers select the recent official reference. Their value differs in how much operational verification they add.

Evaluation target

Strong answers use 100 requests per minute as the current documented default, explain why the primary and recent source controls, and still verify account-specific limits before launch.

Reusable comparison method
  1. Prefer the current primary source.
  2. Treat older secondary and unverified sources as weaker evidence.
  3. Verify account and endpoint-specific limits before launch.

Three captured outputs under one prompt and parameter set; reviewed Aug 27, 2026. This experiment describes these runs only and is not a global model ranking.

Build the next step

Turn this result into a reusable method.

A source-priority rule plus an operational verification step.

Learn the review method