How to run a growth experiment step by step?

Write the hypothesis, choose your metrics before launch, size the sample, run pre-launch checks, run the test without touching it, read it with a sample ratio check first, then write it up. Keep the steps in that order every time. Most of the damage in testing happens when a team skips straight from launch to reading the result.

What is a good experiment hypothesis template?

Use the common format from Optimizely's documentation: If [cause], then [effect], because [rationale]. The cause is the change you ship, the effect is the metric you expect to move, and the rationale is the evidence behind your expectation. Write the problem first, add the baseline for the metric, and describe the change in words so a reader understands it without screenshots.

What are guardrail metrics in A/B testing?

Guardrail metrics are the metrics you are trying not to hurt, as GrowthBook's documentation puts it. They catch damage in other parts of the business, such as support tickets, page load time or early churn, while your one primary metric decides the ship call. Keep a fixed list of three to five guardrails that every test reports against, with a defined direction of harm for each.

How to tell if an A/B test result is statistically significant?

Read the primary metric once, at the stop date fixed before launch, because peeking more often raises the false positive rate. Correct for multiple metrics: testing 20 metrics at a 5% significance level gives around a 64% chance of at least one false win, so a Bonferroni correction divides 5% by 20 to give 0.25% per secondary metric. Then check the confidence interval: if zero sits outside it, the lift is statistically significant, but compare it against the smallest lift worth shipping before you decide.

What to include in an experiment write-up?

Record the experiment name, owner, run dates, hypothesis, primary metric, guardrail metrics and design: unit of assignment, split, planned sample size and duration. Add the checks passed, the primary metric effect with its confidence interval, each guardrail, the decision (roll out, discuss or do not roll out) and whether the decision rule was followed. Close with what you learned and the next test.

How to run a growth experiment end to end, with a write-up template you can copy

Q: What is sample ratio mismatch?

Sample ratio mismatch (SRM) is a statistically significant difference between the ratio of users counted in each variant and the split configured before the test began. It means some users are missing from one arm, and missing users are rarely random. Microsoft requires every A/B test to pass an SRM check at p-value < 0.0005 before anyone analyzes its effects, using a chi-square test on user counts.

To run a growth experiment, follow the same seven steps in order: write the hypothesis, choose metrics before launch, size the sample, run pre-launch checks, run it untouched, read it with a sample ratio check first, and write it up. A recent Microsoft analysis found about 6% of its A/B tests had a sample ratio mismatch, which is why that check comes before any result. Plan at least one to two weeks of run time.

How do you run a growth experiment?

Run every growth experiment as the same seven steps, in the same order. In our reading, most of the damage in testing happens when a team skips straight from launch to reading the result.

Write the hypothesis. State the change, the metric you expect it to move, and why. In Matt Lerner's sprint process, published by First Round, "every sound hypothesis for a growth experiment must start with a customer insight." Score the idea against the rest of your queue with an ICE score and log it in the experiment backlog template.
Choose the metrics before launch. Name one primary metric tied to your north star metric. Then add the metrics that tell you if something broke. Microsoft Research says a core set should contain user satisfaction, guardrail, feature and engagement, and data quality metrics.
Size the sample. A power calculation sets a lower bound on the number of units the test directly affects. Run your numbers through the sample size calculator and write the required count down.
Run pre-launch checks. Confirm the randomization unit fits the test. Cookie-based IDs are unstable, so Microsoft advises using them only for short-term experiments.
Run it without touching it. Hold the split you configured. Ramping variants up unevenly can cause a sample ratio mismatch on its own.
Read it, sample ratio check first. At Microsoft, every A/B test must first pass a Sample Ratio Mismatch (SRM) test before anyone analyzes its effects. The threshold is p-value < 0.0005, and a recent analysis found about 6% of Microsoft's tests had one.
Write it up. Record the result, the reason, and the next test. In Lerner's words, most growth experiments fail, so the learning is the main output.

A worked example of step 6

Say you split 10,000 visitors 50/50. Do not judge the split by looking at the two counts. Microsoft notes that the ratio alone lacks sample size information, so use a chi-square statistic. If the check fails, stop and find the cause before reading the primary metric.

The stakes are real. An MSN test raised carousel cards from 12 (A) to 16 (B) and first showed lower engagement. It had failed the SRM check: engaged users in B had tripped a bot filter and been dropped. Once that was fixed, the results were flipped.

We treat steps 4 and 6 as non-negotiable. In our reading, they cost an hour and decide whether the other five steps meant anything.

How do I write a good experiment hypothesis?

A hypothesis is a prediction written before the test starts, in one fixed sentence. Optimizely's documentation gives the common format: If [cause], then [effect], because [rationale]. The cause is the change you ship. The effect is the metric you expect to move. The rationale is the evidence that makes you expect it.

The same guide says strong hypotheses have three parts: a definition of the problem, a proposed solution, and a result. Write the problem first. If the solution fails, the problem statement still stands, and you can test the next solution against it.

A worked example: removing funnel links

If we remove the navigation links from the checkout pages, then the share of visitors who complete a purchase will rise above the current 5 in 100, because visitors who start checkout leave through those links before they pay.

Each part can be checked. The cause is a single change. The effect has a baseline. The rationale names behavior you can confirm in your analytics before you launch.

Describe the change in words, so a reader understands it without screenshots. A designer who joins later should be able to rebuild the variant from the sentence alone.

Fill-in template

Copy this into your experiment backlog:

Problem: [what users struggle with, in their terms]
Evidence: [the metric or research that shows it]
Hypothesis: If [change], then [metric] will [rise or fall] from [baseline], because [rationale].
Primary metric: [one metric that decides the test]
Success also looks like: [downstream metrics that should follow]

The brief: one record per test

In Optimizely's Collaboration tool, a hypothesis captures all the requirements, workflow, collaboration, data, and analysis for an experiment in one place. Brief fields work as metadata, and teams reuse them to filter through hypotheses and activities. You can follow the same pattern in a shared doc or spreadsheet.

Brief field	What to record before launch (our suggested set)
Hypothesis	The full if, then, because sentence
Owner and reviewers	Who builds, who approves, who analyzes
Variants	A plain description of control and each variant
Metrics	Primary metric, guardrails, baseline
Run dates	Planned start and stop
Results	Left blank until the test ends

Keep the results field in the same brief. When the test ends, the prediction and the outcome sit side by side, and anyone can see whether the rationale held.

What are primary and guardrail metrics?

Every experiment carries three kinds of metric, and you choose all of them before launch. In GrowthBook's documentation, goal metrics are the ones you are trying to improve, secondary metrics help you learn about impacts but are not primary objectives, and guardrail metrics are "metrics you’re trying not to hurt."

One primary metric decides the ship call

Pick a single primary metric, the number the test is built to move. The same documentation notes these are often called the OEC, for Overall Evaluation Criterion. Fix it in the brief before the test starts. If you choose it after the results come in, you will be tempted to pick whichever number moved the way you hoped.

Keep the list short. Loading every metric you track into one test can lead to false positives caused by random variations. Secondary metrics belong in the analysis as context. They should not decide whether you ship.

Guardrails catch damage somewhere else

A change can lift activation and quietly hurt something two teams away. A standing guardrail collection guards against that. Statsig's metrics catalog lets you create a collection of guardrail metrics that teams check in every experiment to detect unexpected effects in other parts of the business. The same principle works in a spreadsheet: one fixed list that every test reports against, whoever runs it.

Statsig's decision framework uses this pairing directly. You select one primary metric and one or more guardrail metrics, and the tool recommends a ship action from how they move. It highlights recommended actions but doesn't enforce them, so the call stays with your team.

Pairings to start from

Primary metric	Guardrails to watch (our reading)	Why it can break (our reading)
Activation rate	Support tickets, page load time	A new onboarding step can confuse users or add weight to the page
Trial-to-paid conversion	Early churn, refund requests	Pushing a faster upgrade can bring in buyers who leave in the first month
Signup rate	Activation rate, page load time	More signups of lower intent can dilute later steps
Checkout conversion	Support tickets, churn	Removing friction can hide pricing terms that customers later dispute

Compare churn guardrails against your own baseline. Our monthly churn benchmarks give outside context.

Set it up this week

Write down three to five guardrails that every test in your company reports, whatever it targets.
For each, define the direction that counts as harm and how far it may move before someone reviews the test.
Add the list to the experiment brief template so nobody has to remember it.
In the readout, report the primary metric first, then every guardrail, including the ones that did not move.

How big a sample and how long a run do you need?

Fix the sample size and the run length before launch, then hold to both. In our reading, the input that moves the sample size most is the effect you want to detect. GrowthBook's documentation puts it plainly: "If you think the experiment will have a large effect, the smaller total number of events you need to collect." Its experiments guide adds that "experiment power and MDE are important to predict how many samples are required," where MDE is the minimum detectable effect.

Enter your baseline conversion rate and your MDE into our sample size calculator. Write the per-arm number in the brief.

Turn the sample into a run length

GrowthBook recommends running "for at least 1 or 2 weeks to capture variations in your traffic." Its tooling can also hide results until a minimum metric total is reached, which helps you avoid premature peeking.

Here is a worked example, not a benchmark. Say 10000 users a week reach the pricing page and you expose 20% of them, so 2000 users a week enter the test. With a 50/50 split, those 2000 weekly users give each arm 1000 a week. If the calculator returns 10000 users per arm, 10000 divided by 1000 a week means a 10 week run. If that is too slow, raising exposure to 50% of those 10000 weekly users puts 5000 a week into the test. In our reading, a shorter run still needs at least 14 days.

Choose a split and a hashing attribute

GrowthBook assigns users with deterministic hashing: each user's hashing attribute, usually user ID, and the experiment name are hashed together into a number from 0 to 1. The same inputs always give the same number. Pick an attribute that stays stable for the whole run, such as a logged-in user ID. If you have to change the settings mid-test, GrowthBook offers sticky bucketing, which requires additional configuration.

De-risk with exposure, not uneven splits

The usual way teams de-risk is to give 80% of users the control and 20% the new variation. GrowthBook advises keeping splits equal and lowering overall exposure instead: 20% exposure at 50/50, so each variation gets 10%. That way you can ramp exposure up or down without any user switching variations. This also sidesteps the uneven ramp-up problem covered earlier.

Before the real test, run an A/A test to confirm traffic splits correctly.

Setting	What to record	Worked example value
Hashing attribute	Stable ID used for assignment	User ID
Overall exposure	Share of eligible users in the test	20%
Split	Share per arm	50/50
Sample per arm	Number from the calculator	10000
Planned run	Sample divided by weekly users per arm	10 weeks

Run these checks before you launch

A test that launches broken wastes every day it runs. Microsoft's experimentation team sorts its trust patterns into three major categories across the test's lifecycle, starting with the pre-experiment stage. Treat the table below as a gate. Nobody turns the flag on until every row has an owner's initials next to it.

Check	What to confirm before launch	Who signs off (our suggestion)
Randomization unit	The ID matches the hypothesis. Cookie IDs are unstable because users can clear their cookies and switch variants, so keep them to short tests.	Growth lead
Assignment method	Assignment hashes a stable attribute, and that attribute exists when the code runs. The hashing attribute may not be available until tracking has loaded.	Engineer
Server-side or client-side	Server-side assignment avoids any issues with flickering. If you stay client-side, load the SDK higher in the code or use inline experiments.	Engineer
Exposure point	Users enter the test as close to the actual treatment exposure as possible, not on first page load.	Growth lead
Event tracking QA	You fired every primary and guardrail event yourself in each variant, and each one reached the warehouse with the right variant label.	Analyst
A/A or dry run	The A/A test from the sample size section ran clean, and the observed split matches the configured split.	Analyst
Alerts	The expected ratio is set, and alerts fire on a split mismatch and on a sharp drop in a key metric.	Analyst

Why the ID and alert rows matter

Microsoft's own incident shows how a fixed bug can still break a test. In 2020, a product bug gave a subset of users duplicate Device IDs, the randomization unit for its A/B tests. The bug was fixed and those users received new IDs.

The rule we take from it: after any change to how IDs are created or stored, rerun the dry run before you trust a live test.

For the alert row, Microsoft configures the ratio between groups before each experiment, typically 1:1 split. As an example, not a benchmark: with 10000 users and a 50 percent split, each arm should hold about 5000. Pick one key metric, write down the drop that would make you stop the test, and send the alert to the person who owns the test.

Its team suggests a couple hundred seeds.

What is sample ratio mismatch and what do you do about it?

Sample ratio mismatch (SRM) is what Microsoft Research describes as a statistically significant difference between the ratio of users counted in variants A and B from what was configured before the experiment began. If you set a 50/50 split and the counts drift too far apart for chance to explain, some users are missing from one arm.

The same article compares it to survivorship bias in World War II aircraft analysis, where the planes that never returned needed to be included in the analysis for its results to be trustworthy. Its MSN example shows the cost. A test in which the number of rotating cards was increased from 12 (A) to 16 (B) first read as a loss. The cause was that users in B engaged with it enough to confuse a bot detection algorithm which then filtered them out of the analysis. The lesson the authors draw is that missing users are rarely just some users.

SRM is not rare. Microsoft reports that about 6% of A/B tests have an SRM.

Check it with a chi-square test

Eyeballing the split does not work, because the ratio lacks information about the sample size. Use a chi-square goodness-of-fit test on user counts before you look at any metric.

Here is a worked example, not a benchmark. A 50/50 test enrolls 10000 users, so you expect 10000 divided by 2, or 5000, per arm. Control logs 5000 plus 50, or 5050, and treatment logs 5000 minus 50, or 4950. For each arm, square the gap and divide by the expected count: 50 times 50 is 2500, and 2500 divided by 5000 is 0.5. Add the two arms: 0.5 plus 0.5 gives a statistic of 1, with one degree of freedom.

Compare the resulting p-value with a strict cutoff. Microsoft's is p-value < 0.0005. A statistic of 1 lands far above it, so this split counts as noise.

Trace the root cause before trusting anything

Microsoft's taxonomy sorts causes by stage. Start by cutting the counts by segment, because an SRM root-cause is localized to a particular segment more often than not.

Stage	Example causes	First thing to check (our reading)
Assignment	Incorrect bucketing, faulty user IDs, carryover effects	Hashing key and ID stability
Execution	Redirecting users in one variant, variant changing engagement	Bot filters, redirects, load failures
Log processing	Incorrect data joins	Join keys between exposure and event tables
Analysis	Biased conditions used to segment the analysis	Filters applied after assignment

Also check whether someone was ramping up the experiment variants unevenly. Log the cause in your write-up. Fix the cause, restart the test and discard the old readout.

How do you know if your A/B test result is real?

A result is real when it survives three checks: you read it at the planned stop, you corrected for the number of metrics you looked at, and the interval clears the smallest lift worth shipping. Do them in that order.

Fix the stop date before launch

Write the stop date into the experiment doc next to the sample size from your power calculation, and read the primary metric once, on that date. GrowthBook's documentation explains that the more often an experiment is peeked at, the higher the false positive rate, and it advises using a predetermined sample size or duration and sticking to it. Daily dashboards are fine for guardrails and bugs. They are not a decision point. If you need to stop early as a routine, use a sequential testing method built for it.

Correct for every metric you checked

If you test at a 5% level of significance for 20 different metrics, the chance of at least one false win is around 64%. The named fixes include the Bonferroni correction, False Discovery Rate (FDR) correction, or the Benjamini-Hochberg procedure. A simple way we apply the first: divide the 5% threshold by the 20 metrics, so each secondary metric needs a p-value under 0.25%. Your one primary metric, named in advance, keeps the plain 5%. A surprise win on a secondary metric becomes the hypothesis for a follow-up test, not a ship decision.

Read the interval against the lift you need

Practical significance is a separate question. Say you set 1 point as the smallest activation lift worth the maintenance cost.

Interval vs. your minimum lift	Decision (our reading)
Whole interval above the minimum	Ship
Above zero, straddles the minimum	Discuss: ship if cheap and guardrails hold, or rerun larger
Includes zero	Do not ship

Statsig's decision framework uses three similar labels: Rollout Winning Group, Discuss and Do Not Roll Out. A small copy change with flat guardrails ships. A new onboarding flow that needs ongoing work gets a rerun with a larger sample.

The experiment write-up template and the ship decision

A test is not finished when the dashboard turns green. It is finished when someone has written down what happened and what the team will do next. In Matt Lerner's sprint model, most growth experiments fail, so the primary outcome is learning. The write-up is where that learning gets kept.

The template

Copy this into your doc tool or your growth experiment backlog. Fill every field before anyone discusses the result.

Experiment name:
Owner: Run dates:
Hypothesis: Because [customer insight], we believe [change]
will move [primary metric] for [segment].
Primary metric:
Guardrail metrics:
Design: unit of assignment, split, planned sample size, planned duration
Checks passed: sample ratio check, exposure point, no mid-test changes
Result: primary metric effect with its confidence interval
Guardrails: each one, moved or flat, with interval
Decision: roll out / discuss / do not roll out
Rule followed? yes / no. If no, who signed off and why:
What we learned:
Next test:

Optimizely's documentation suggests that teams dedicate a section in the brief to capturing the post-experiment analysis.

Mapping results to the decision

Result pattern	Decision (our reading)	Sign-off needed (our reading)
Primary metric up, interval clear of zero, guardrails flat	Roll out	Owner
Primary up but a guardrail down, or interval crosses zero	Discuss	Owner plus one reviewer
Primary flat or down	Do not roll out	Owner

Write the rule into the template before launch, then make breaking it expensive. Statsig's decision framework highlights recommended actions but doesn't enforce them. Its settings let a team require reviews when decision opposes recommended decision. Under that setting, a review is triggered by shipping a treatment group that isn't recommended, and in the discuss case by shipping any treatment group. You can copy the same rule in a spreadsheet. Any override gets a named reviewer and one sentence of reasoning in the doc.

Run one this week

Monday: pick the top idea from your backlog and write the hypothesis, metrics and design fields.
Tuesday: run the pre-launch checks and launch.
Let it run to the sample size or duration you fixed in advance. As an example, a 14-day run spans two sprint reviews.
At the review after it ends: fill in the result, apply the table, record sign-off if the call goes against the rule.
Same meeting: write the next test, so the loop restarts.

The cadence matters less than its regularity. Lerner notes that some teams have growth meetings on Tuesdays, some on Fridays, and some every two weeks. We think the useful rule is that no review closes without a filled template for every test that ended.

What this page does not cover

This page does not cover sequential testing methods in detail, how to choose or set up an experimentation platform, or how to calculate confidence intervals by hand. Its worked numbers are examples, not benchmarks, so run your own baseline through a sample size calculator before you launch.