The one-page experiment write-up: hypothesis, design, result, decision
One page per experiment, written in two halves: the design before the test starts (hypothesis, metric, sample, duration, what would count as a win) and the result after it ends (the numbers with their interval, the decision, what you learned). Writing the first half before launch is what stops a team from reading the result it hoped for. Copy it or download it as Markdown.
How to use it
Fill in the design half and get it signed off before the test goes live. Do not change the metric or the sample once it is running.
Write the result half within two days of the test ending, while the context is fresh, and link it from the backlog row.
Before the test
- Hypothesis
- We believe [change] for [audience] will [move metric] because [evidence].
- Primary metric
- [The one number that decides the result, and where it is measured.]
- Guardrail metrics
- [Numbers that must not get worse, such as refunds, unsubscribes or support tickets.]
- Baseline and smallest lift worth finding
- [Baseline rate] and [lift], with the reason that lift matters to the business.
- Sample and duration
- [Visitors per variant] and [days, rounded up to whole weeks], from the sample size calculator.
- What counts as a win
- [The result that would make you ship, and the result that would make you drop it.]
- Sign-off
- [Name, date.]
After the test
- What ran
- [Dates, traffic, anything that went wrong: a bug, a campaign, an outage.]
- Result
- [Control rate], [variant rate], [relative change] with its [confidence interval], and whether it met the win condition.
- Guardrails
- [Each guardrail, and whether it held.]
- Decision
- [Ship, drop or rerun], and why.
- What we learned
- [What this changes about the other ideas in the backlog. Re-score them.]