Short answer
Run an A/B test until each variant reaches the sample size you worked out before starting, and for at least two full weeks so every weekday is covered. The sample size depends on your baseline conversion rate, the smallest lift you care about, and the confidence and power you choose (usually 95% and 80%). Don't stop the test early because one variant looks like it's winning.
Key takeaways
- Decide the sample size before the test starts, and don't stop early.
- The smaller the lift you want to detect, the more traffic you need: half the lift, about four times the visitors.
- Run for whole weeks, at least two, to cover weekday and weekend behavior.
- More variants means more traffic. With low traffic, test one bold change.
- If the calculator says months, test a bigger change or use a before-and-after comparison.
"How long should we run this test?" is the question I get most from teams new to A/B testing. The honest answer is: until you have enough visitors to trust the result, which you can work out before you start. Put your numbers in the calculator below.
Free calculator
A/B test sample size and duration
What each input means
- Current conversion rate: how often visitors to this page convert today. Use the last 4–8 weeks.
- Smallest lift to detect (minimum detectable effect): the smallest relative improvement worth knowing about. 10% means going from 3% to 3.3%.
- Confidence: how sure you want to be that a difference isn't luck. 95% is the usual choice.
- Power: the chance of detecting a real lift of that size if it exists. 80% is standard.
Why small lifts need so much traffic
Visitors needed per variant
Baseline conversion rate 3%, 95% confidence, 80% power
Show the numbers as a table
| Lift to detect | Visitors per variant |
|---|---|
| +5% | 207,940 |
| +10% | 53,211 |
| +15% | 24,193 |
| +20% | 13,914 |
| +25% | 9,100 |
| +30% | 6,454 |
This is the trade-off that decides what you should test. With modest traffic, you can only reliably detect big lifts, so you need bold changes: a new headline and offer, a different page structure, a quiz instead of a form. Changing a button color won't produce a lift you can measure.
Five mistakes that break A/B tests
- Peeking and stopping early. Checking every day and stopping when it "looks significant" inflates false positives dramatically. Evan Miller's article "How Not To Run an A/B Test" explains why. Decide the sample size first.
- Running partial weeks. People behave differently on Monday morning and Saturday night. Always run whole weeks.
- Too many variants. Each extra variant needs its own full sample. Three variants with little traffic means months of testing.
- Ignoring sample ratio mismatch. If you split 50/50 but one variant got noticeably more visitors, something's broken in the setup. Check before you read the result.
- Testing during unusual periods. Sales, launches and holidays change who visits. Avoid them, or run long enough to include normal weeks.
What to do with low traffic
If the calculator says your test needs three months, don't run it. Instead:
- Test bigger changes that could plausibly move conversion 20% or more.
- Test where the traffic is: the product page template or the homepage, not a single low-traffic page.
- Use an earlier metric, like add-to-cart instead of purchase, which happens more often.
- Research first, then ship: with strong evidence from recordings, reviews and surveys, make the change and compare the weeks before and after.
Want test ideas worth the traffic? That's what A/B test design is for.
FAQ
How long should an A/B test run?
Until each variant reaches the sample size you calculated before starting, and for at least two full weeks. For most sites that's two to six weeks.
What sample size do I need for an A/B test?
It depends on your baseline conversion rate and the smallest lift you want to detect. At a 3% conversion rate, detecting a 10% lift needs roughly 53,000 visitors per variant at 95% confidence and 80% power.
Can I stop an A/B test when it reaches significance?
Not if you're checking repeatedly. Stopping as soon as significance appears inflates the chance of a false winner. Wait for the planned sample size.
What is a minimum detectable effect?
The smallest relative improvement your test is designed to detect reliably. A smaller minimum detectable effect needs much more traffic.
Sources
Conversion designer. I design and build product pages, landing pages and websites, A/B test them and show what each change earned. CXL certified in Conversion Optimization and Ecommerce Marketing, and a guest speaker at NYU School of Professional Studies.
