How Much Traffic Do You Need for A/B Testing? (And What to Do If You Don't Have It)
A/B testing is the thing everyone tells you to do, and for most small sites it is the wrong advice. Not because testing doesn't work. Because you don't have the traffic to finish a test before the answer stops mattering.
Nobody runs this math before installing a testing tool. Let's run it.
The number, and why it's so big
There's a standard rule of thumb for how many visitors an A/B test needs per variant, at the usual settings of 95% confidence and 80% power. Plug in a 2% baseline conversion rate, which is normal for a small store, and here's what you get.
Is your site converting as well as it could be?
Get a CRO Audit - $99| Lift you want to detect | Per variant | Total visitors |
|---|---|---|
| 50% (2.0% to 3.0%) | ~3,100 | ~6,300 |
| 20% (2.0% to 2.4%) | ~19,600 | ~39,200 |
| 10% (2.0% to 2.2%) | ~78,400 | ~156,800 |
Look at what happens between those rows. Halving the lift you're chasing roughly quadruples the traffic you need, because sample size scales with the inverse square of the effect. Smaller wins are exponentially more expensive to prove.
Now put your own traffic against it. At 500 visitors a month, detecting a 20% lift takes about 78 months. That's over six years, on one test. At 2,000 visitors a month it's still around 20 months. Your business will change, your traffic mix will change, and the seasons will turn twice before that test calls a winner.
And a 20% relative lift is not a small ask. Most real tests produce less.
What people do instead, and why it's worse than not testing
Faced with a test that won't finish, owners peek. They watch the dashboard, see variant B pull ahead in week two, call it, and ship.
The problem is that early in a test the numbers swing wildly for the same reason a coin can come up heads six times running. Stopping the moment you see a lead means you'll declare a winner nearly every time, and a good share of those winners are noise. You then implement the change, your conversion rate doesn't move, and you conclude that CRO doesn't work.
An underpowered test isn't a weaker version of a real test. It's a random number generator that makes you feel rigorous. Not testing at all is genuinely better than testing badly, because at least then you know you're using judgment.
What to do below the threshold
Small sites are not stuck. They just have to use methods that don't need statistical significance.
Fix the things that don't need proof. A headline that doesn't say what you sell. A checkout that requires account creation. A form asking for a phone number you never call. Nobody needs a test to tell them a broken link is bad. Most small sites have a stack of these, and testing them would be like running an experiment on whether the lights work.
Watch session recordings. Twenty recordings of real people using your site will teach you more than an underpowered test, and tools like Microsoft Clarity are free. You're not measuring, you're observing. Watch where people hesitate, where they scroll back up, where they rage-click something that isn't a button.
Test with five people, not five thousand. Hand your site to five people who match your customer and ask them to complete a purchase while talking out loud. Usability problems cluster hard. The same few issues surface again and again, and you'll have your list by person three. This is qualitative work and it does not need a sample size.
Make changes sequentially and watch the trend. Change one meaningful thing, give it a few weeks, and look at the direction. This is not clean science. Seasonality and traffic mix will muddy it. But over a year of deliberate sequential changes you will know whether your site is better, and you'll have shipped a dozen improvements instead of finishing one test.
Talk to people who didn't buy. Email a handful of visitors who abandoned, or customers who took a long time to decide, and ask what nearly stopped them. The answers are usually blunt and specific, and no amount of traffic would have surfaced them.
When you've earned A/B testing
The rough line is a few thousand conversions a year, not visits. Once you have enough conversion volume that a test can resolve inside a month or two, testing becomes the right tool and it gets genuinely powerful. Below that, the honest answer is that you should be fixing obvious problems and shipping, not measuring.
There's a version of this advice that sounds like an excuse for sloppiness. It isn't. The discipline just moves. Instead of proving each change statistically, you commit to looking hard at the site, ranking what's wrong, fixing the top items, and being honest about what you can and can't know.
That ranked list is the whole job at this size. Here is what one looks like on a real site. It's also what I do for $99, and what tends to be on it.