Quick answer
To read A/B test results, look at two numbers together. Lift is how much better or worse the variant did than the original, as a percentage. Confidence is how sure you can be that the difference is real rather than chance. A big lift with low confidence is not a winner yet. Decide the run time in advance, usually at least one full week, and do not stop early because one version looks ahead. When the test ends, act on clear wins, keep the original when there is no real difference, and record what you learned either way.
Key takeaways
- Lift shows the size of the difference; confidence shows whether to trust it
- A large lift on few conversions is often just chance
- Run for at least one full week to cover weekday patterns
- Set the end point before launch and stick to it
- Peeking and stopping early is the most common cause of false winners
- No clear difference is a valid, useful result
- Check test health first: is traffic splitting and are conversions recording?
📘 Main article: This guide supports Scale.gg AB Lincoln: A/B Split Testing That Measures What Pays. There we show AB Lincoln reporting lift, confidence, attribution and health with a plain-English recommendation. This guide explains what those numbers mean and how to act on them.
The moment a test starts, it is tempting to check it every hour. On day two, version B is ahead by 40%. Should you call it? Almost certainly not, and knowing why is the difference between learning something and fooling yourself.
This guide answers one question: how do you read A/B test results, and when is it safe to stop? It pairs with our main article, Scale.gg AB Lincoln: A/B Split Testing That Measures What Pays, and our AB Lincoln video on the Prodify Digital channel.
What does lift mean in an A/B test?
Lift is the percentage difference between the variant and the original on your chosen goal. If the original converts 2.0% of visitors and the variant 2.5%, the lift is 25%, because 0.5 is a quarter of 2.0.
That example also shows why lift can sound bigger than it is. A 25% lift here means half a percentage point more visitors converting. Both ways of describing it are correct, so be clear which one you mean when you report a result.
| Original rate | Variant rate | Lift |
|---|---|---|
| 2.0% | 2.5% | +25% |
| 10% | 11% | +10% |
| 5% | 4% | −20% |
These are illustrative figures to show the arithmetic, not typical results.
What does confidence mean?
Confidence tells you how likely it is that the difference you see is real, not random noise. Visitors vary, and two identical pages will rarely produce exactly the same numbers. Confidence measures whether the gap is bigger than that natural wobble.
Many testers use 95% as a common threshold before acting, though it is a convention rather than a law. Our AB Lincoln video does not say which threshold or statistical method it uses, so check the tool’s own documentation.
🔎 Unconfirmed / rumour: Our video confirms that AB Lincoln reports confidence and a plain-English recommendation, but not the method behind it or the level at which it recommends a winner. Treat the recommendation as guidance and check the underlying numbers.
How many conversions do you need?
More than most people expect, especially for small differences. The smaller the lift you want to detect, the more conversions you need. As a rough rule of thumb, a handful of conversions per version is nowhere near enough, and even a few dozen can be shaky when the difference is small.
- Big change, steady traffic: you may get a clear answer in a week or two
- Small change, low traffic: you may never get a clear answer, so test something bolder
- Very low traffic: consider a goal higher up the funnel, as covered in our sibling guide How to Choose the Right Conversion Goal for an A/B Test
A free online sample size calculator can give you a target before you start. Use your current conversion rate and the smallest lift that would be worth acting on.
When should you stop an A/B test?
Stop when you reach the run time or sample size you set before launch, not when the result looks good. In our AB Lincoln video, you choose how long the test runs as part of the setup.
| Stop when… | Do not stop when… |
|---|---|
| The planned end date or sample size is reached | One version jumps ahead on day two |
| At least one full week has passed | You have had one unusually good day |
| Health checks show tracking is working | You are bored of waiting |
| Something is broken and the test is invalid | The result disagrees with your gut feeling |
⚠️ Warning: Checking results repeatedly and stopping the first time one version looks ahead greatly increases the chance of a false winner. If you must look, look, but do not act until the planned end.
What should you do with each kind of result?
- Clear win, healthy test: roll out the variant, check your guardrail measure, and plan the next test
- Clear loss: keep the original and note why your hypothesis may have been wrong
- No clear difference: keep whichever is simpler or cheaper, and test a bolder change next
- Unhealthy test: fix tracking or traffic splitting and start again; do not trust the numbers
Write each result into one place. AB Lincoln keeps every test in one list, as shown in our main article. If you test emails too, Keep an Email Experiment Log You Can Actually Learn From shows a simple log format.
What mistakes lead to false winners?
- Stopping early on a lucky streak
- Testing several changes at once and not knowing which one mattered
- Running during an unusual week, such as a big sale or holiday, and treating it as normal
- Ignoring health warnings about uneven traffic or missing conversions
- Reporting lift without confidence, so a lucky result sounds like a fact
Want the full picture? Read Scale.gg AB Lincoln: A/B Split Testing That Measures What Pays, and see our Starter Bonus Bundle for Scale.gg.
Frequently asked questions
What is a good lift in an A/B test?
Any lift you can trust that is big enough to matter for your business. A small, reliable lift beats a big one with low confidence.
What confidence level should I use?
Many testers use 95% as a common threshold. Whatever you choose, decide it before the test starts.
How long should an A/B test run?
At least one full week, and until you reach the sample size you planned. Low-traffic pages may need several weeks.
Is a test with no winner a waste of time?
No. It tells you that change did not matter much, which saves you from rolling out something on a hunch.
What does test health mean in AB Lincoln?
Our video lists health alongside lift, confidence and attribution. In general it means checking traffic is splitting and conversions are recording. See our AB Lincoln article.
The bottom line
Reading A/B test results well comes down to pairing lift with confidence, checking health, and stopping only at the end point you set in advance. Act on clear wins, keep the original when there is no real difference, and record every outcome. That habit turns a series of tests into real knowledge about your audience.
Related posts
- Scale.gg AB Lincoln: A/B Split Testing That Measures What Pays
- How to Choose the Right Conversion Goal for an A/B Test
- Keep an Email Experiment Log You Can Actually Learn From
