A/B testing your website: improve conversion with data, not gut feelings

Almost every meeting about a website ends up in the same argument: the button should be green, the headline is too long, the form asks for too much. Everyone has an opinion, usually defended with great confidence, and in the end the opinion of whoever is most senior or most persistent wins. The result is a website built on gut feelings.

A/B tests exist precisely to settle those arguments. Instead of guessing which version works best, you show each one to a share of your visitors and let their actions decide. In this article I explain exactly what an A/B test is, what's worth testing, how much traffic you need to trust the result, the mistakes that invalidate it and which tools you can use today, now that Google Optimize no longer exists.

An A/B test is a controlled experiment: you take a page or an element of your website (version A, the current one) and create a variant with one specific change (version B). Then you split traffic randomly between the two and measure which one gets more conversions, whether that's forms submitted, bookings, purchases or clicks on a button. Because the split is random and simultaneous, the differences you see are down to the change and not to the time of year, the day of the week or a one-off campaign.

It's worth distinguishing it from things it's often confused with:

• Comparing before and after is not an A/B test. If you change the headline in October and sales go up, you don't know whether it was the headline, the season or a press mention.
• A multivariate test tries several combinations of elements at once. It's powerful, but it needs far more traffic, so it rarely makes sense for a small business.
• A split URL test compares two completely different pages. It's useful for major redesigns and follows the same logic as an A/B test.

The most common mistake is running tests for the sake of it: “let's try a red button and see what happens.” A good test starts with a hypothesis based on something you've observed. For example: “In analytics I can see that many users abandon the quote form at the phone number field. I think that if we make that field optional, submissions will increase, because people don't want sales calls.”

That hypothesis has three parts: an observed problem, a specific change and a reason. And it comes with a primary metric that will decide the outcome (in the example, forms submitted). Choose a single metric before you start. If at the end you search through ten metrics for the one that did best, you'll almost always find one, even if purely by chance.

To have those starting observations you need properly configured analytics with conversions defined. I won't repeat how to do it here; you'll find it step by step in my guide to Google Analytics 4 for small businesses.

Not everything deserves a test. The changes most likely to move conversion are the ones that affect the user's decision, not cosmetic details. These are the areas where I usually focus experiments:

Headlines and value proposition

The headline is the first thing people read and it decides whether they stay or leave. You can test a benefit-led angle against a problem-led one, a more specific message against a more general one, or including a verifiable fact about your service (delivery time, service area, guarantee) against leaving it out.

Buttons and calls to action

More than colour, it's usually the wording and position that matter. “Submit” says very little; “Get a free quote” or “See available times” explains what will happen on click. You can also test repeating the call to action at different points on a long page, or highlighting a single action instead of several.

Forms

Forms are one of the best places to experiment because every field is a small obstacle. Try removing fields, making others optional, splitting a long form into steps or changing the text next to the submit button. Just don't sacrifice lead quality: if removing a field brings more submissions but worse customers, you haven't gained anything.

Prices and how you present them

Tread carefully here. Testing how you present prices (the order of plans, which plan is highlighted, monthly versus annual pricing, showing an indicative price or not) is common and legitimate. Showing different prices for the same product to different customers is another matter: it can damage trust if discovered, and you should check the consumer protection rules that apply to your case first.

This is where many A/B tests fall apart. To know whether the difference between A and B is real and not down to chance, you need enough visits and conversions. That number depends on three things: your current conversion rate, the size of the improvement you want to detect and the confidence level you require (95% is the usual standard).

Here's an example with round numbers to give you an idea: if your page converts at 2% and you want to reliably detect a variant that lifts it to 3%, the standard formula for comparing proportions (at 95% confidence and 80% statistical power) calls for around 3,800 visits per variant, so roughly 7,600 in total. And that's to detect a large improvement, 50% in relative terms. To detect smaller improvements, the figure multiplies.

The practical takeaways for a small business are clear:

• If your website gets little traffic, test big changes (a different value proposition, a much shorter form), not nuances. Nuances need huge samples.
• Calculate the sample size before you start using an A/B test calculator, and estimate how many weeks you'll need with your real traffic.
• If the calculation says you need a year, that test doesn't make sense on your website. Look for another method instead (I cover some below).
• Test on the pages with the most traffic and closest to conversion: your flagship service page or the final step of a purchase.

A badly run test is worse than no test at all, because it gives you false confidence. These are the mistakes I see most often:

1. 1Checking results every day and stopping as soon as one is ahead. Early on the numbers swing a lot; if you stop at the first favourable moment, you greatly increase the chance of declaring a false winner. Define the duration or sample size up front and stick to it.
2. 2Changing several things at once in the variant. If B has a different headline, button and image and it wins, you won't know what worked. If you want to test a full redesign, treat it as such.
3. 3Not covering full weeks. Monday behaviour isn't Saturday behaviour. Let the test run for at least one or two full weeks to capture your business's usual cycle.
4. 4Overlapping with events that distort traffic. A one-off campaign, a sale or a media mention can bring visitors very different from your usual ones.
5. 5Ignoring the flicker effect. Many tools load the original page and then modify it with JavaScript, so the user sees a flash of version A before B. That hurts the experience and can bias the result; plus, adding heavy scripts affects your website speed.
6. 6Hunting for winners across segments at the end. “On mobile and on Tuesdays, B wins” is almost always statistical noise if you didn't plan it before starting.
7. 7Not documenting. Record every test: hypothesis, dates, result and decision. A year from now you'll be glad to know what was tested and why.

For years, the default option for many small businesses was Google Optimize, thanks to its integration with Google Analytics. Google announced its closure in early 2023 and, according to its official support page, Optimize and Optimize 360 stopped being available on 30 September 2023. Instead, Google chose to invest in third-party testing integrations with Google Analytics 4. If you come across tutorials still recommending it, they're out of date.

Today the alternatives fall into a few families. I'm not giving prices because they change often and depend on traffic volume; always check each provider's website:

• Commercial experimentation platforms, such as VWO, Optimizely, AB Tasty or Convert. They offer visual editors, targeting and built-in statistics. They're convenient for marketing teams, although their cost usually makes more sense above a certain volume.
• Open-source tools or tools with a product focus, such as GrowthBook or PostHog, which combine experiments with feature flags (switches that turn features on for a share of users). They fit very well if your website is custom-built.
• WordPress plugins, such as Nelio A/B Testing, which let you test pages, headlines or themes from the dashboard itself. Handy if your website runs on WordPress and you don't want to touch code.
• Server-side experiments with custom development. On websites built with Next.js, for example, the variant can be assigned on the server or in middleware before the page reaches the browser. That eliminates flicker and keeps the site fast.

Whatever you choose, make sure the tool respects your website's cookie consent, that conversions are also recorded in your main analytics and that you can remove it easily once the experiment ends. If you're unsure which technology suits your case best, my Next.js vs WordPress comparison explains the differences in approach.

Many small business websites don't have enough traffic for reliable A/B tests within a reasonable time. That doesn't mean you have to decide blindly. There are methods that give you solid clues with far fewer visits:

• Heatmaps and session recordings (with tools such as Microsoft Clarity or Hotjar): you see where real users click, how far they scroll and where they get stuck.
• User testing: ask five people from your target audience to try booking or requesting a quote while thinking out loud. The big problems show up right away.
• Asking your customers: what they hesitated about before hiring you, what almost made them go elsewhere. Those answers are hypotheses ready to apply.
• Proven best practices: some improvements (loading fast, a short form, a clear message) don't need a test to know they help. You'll find the conversion basics in the article on how to build a landing page that converts.

1. 1Review your analytics and find the page with the most traffic that's closest to conversion.
2. 2Observe behaviour with data and, if possible, with heatmaps: where are people dropping off?
3. 3Write a hypothesis with a problem, a change and a reason, and choose a single primary metric.
4. 4Calculate the sample size and duration. If it isn't feasible, switch methods or test a bolder change.
5. 5Launch the test, don't touch it and don't check it daily with the urge to stop it.
6. 6Analyse the result when it ends, roll out the winner and document what you learned.
7. 7Move on to the next hypothesis. Improvement comes from chaining experiments, not from a single one.

A well-designed A/B test turns arguments about your website into questions that can be answered. The key is to start with a data-based hypothesis, choose a single metric, respect the sample size and avoid the mistakes that invalidate results. And if your traffic isn't enough for statistically sound experiments, heatmaps, user testing and asking your customers still let you improve with good judgement.

I'm Elmar Mamedova, a full stack developer based in Málaga, and I like building websites that can be measured and improved, not brochures that get published and forgotten. If you want to get your website ready to experiment without losing speed, or you need help deciding what to test first, tell me about your case and we'll look at it with your data.